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[Name of Document] SPECIFICATION 
[Title of the Invention] 

Signal Processing Device and Signal Processing Method 
[Claims] 
[Claim 1] 

A signal processing device comprising: 

a first signal section detecting means for detecting a first signal section 
from an input signal including at least the first signal section and the remaining 
signal section on a time division basis; 

a first signal extracting means for extracting the signal in the first signal 
section from the input signal in accordance with a result of the detection by the first 
signal section; and 

a recording means for recording the extracted first signal. 
[Claim 2] 

The signal processing device according to claim 1, further comprising a 
characteristic value extracting means for extracting the characteristic values 
characterizing the first signal from the extracted first signal, wherein said recording 
means records the characteristic values. 
[Claim 3] 

The signal processing device according to claim 1 , wherein said first signal 
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section detecting means detects said first signal section from said input signal on 
the basis of a characteristic pattern of the signal appearing in said input signal at 
predetermined time intervals and a characteristic value showing the probability of 
the first signal. 
[Claim 4] 

The signal processing device according to claim 1, wherein said first signal 
section detecting means detects said first signal section on the basis of 
predetermined guide information which is prepared corresponding to said input 
signal. 
[Claim 5] 

The signal processing device according to claim 2, wherein said 
characteristic value characterizing said first signal is amplitude of the signal in the 
first signal section. 
[Claim 6] 

The signal processing device according to claim 2, wherein said 
characteristic value characterizing said first signal is a spectrum of the signal in the 
first signal section. 
[Claim 7] 

The signal processing device according to claim 2, wherein said 
characteristic value characterizing said first signal is a linear prediction coefficient 



2 



of the signal in the first signal section. 
[Claim 8] 

The signal processing device according to claim 2, wherein said 
characteristic value characterizing said first signal is a histogram of a 
predetermined component of the signal in the first signal section. 
[Claim 9] 

The signal processing device according to claim 2, wherein said 
characteristic value characterizing said first signal is mean value of the 
predetermined component of the signal in the first signal section. 
[Claim 10] 

The signal processing device according to claim 2, wherein said 
characteristic value characterizing said first signal is difference of the 
predetermined signal component of the signal in the first signal section. 
[Claim 11] 

The signal processing device according to claim 2, wherein said 
characteristic value characterizing said first signal is the number of changes of the 
state of the signal in the first signal section. 
[Claim 12] 

The signal processing device according to claim 2, wherein said 
characteristic value characterizing said first signal is the time of the change of the 
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state of the signal in the first signal section. 
[Claim 13] 

The signal processing device according to claim 1, further comprising: 
an index information extracting means for extracting information to be used 
as an index representing said recorded first signal; and 

a display means for displaying said extracted index information. 
[Claim 14] 

The signal processing device according to claim 13, wherein said index 
information is an edited signal obtained by editing said first signal. 
[Claim 15] 

The signal processing device according to claim 14, wherein said edited 
signal obtained by editing said first signal comprises a set of signals at the time 
when the state of said first signal changes. 
[Claim 16] 

The signal processing device according to claim 14, wherein said edited 
signal obtained by editing said first signal comprises a signal representing a starting 
part of said first signal and/or a signal representing an ending part of said first 
signal. 
[Claim 17] 

The signal processing device according to claim 13, wherein said index 
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information extracting means extract the signal at a time when the state of said first 
signal changes. 
[Claim 18] 

The signal processing device according to claim 2, further comprising a 
comparing means for comparing characteristic values respectively characterizing 
different first signals. 
[Claim 19] 

The signal processing device according to claim 18, wherein said 
comparing means is detects agreement/disagreement of the first signals in a part of 
the section or in the entire section by comparing said characteristic values. 
[Claim 20] 

The signal processing device according to claim 18, wherein said 
comparing means detects the degree of similarity of the first signals in a part of the 
section or in the entire section by comparing said characteristic values. 
[Claim 21] 

The signal processing device according to claim 18, wherein said 
comparing means performs the comparing operation on a basis of the distance as 
determined by using a predetermined distance scale between vectors formed at least 
one of the amplitude of the signal in the first signal section, the spectrum of the 
signal in the first signal section, the linear prediction coefficient of the signal in the 
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first signal section, the histogram of a predetermined component of the signal in the 
first signal section, the mean value of the predetermined component of the signal in 
the first signal section, the difference in the predetermined signal component of the 
signal in the first signal section, the number of changes in the state of the signal in 
the first signal section and the time of a change in the state of the signal in the first 
signal section. 
[Claim 22] 

The signal processing device according to claim 13, further comprising: 
an index information specifying means for specifying desired index 

information from said displayed plurality of pieces of index information; and 

a retrieving means for retrieving the first signal corresponding to said 

specified index information. 
[Claim 23] 

The signal processing device according to claim 2, further comprising: 
a retrieving means for retrieving the first signal substantially agreeing with 
said first signal from said recording means, using said first signal in a part of the 
section or in the entire section or a characteristic value characterizing the first 
signal as retrieving condition. 
[Claim 24] 

The signal processing device according to claim 2, further comprising: 
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a retrieving means for retrieving the first signal substantially agreeing with 
said first signal from said recording means, using a part or all of said first signal or 
a characteristic value characterizing the first signal as retrieving condition. 
[Claim 25] 

The signal processing device according to claim 1 , further comprising: 
a measuring means for measuring the number of times and/or the hours of 
appearances of a same first signal. 
[Claim 26] 

The signal processing device according to claim 1, further comprising: 
a measuring means for measuring the number of times and/or the hours of 
appearances of similar first signals. 
[Claim 27] 

A signal processing method comprising the steps of: 

detecting a first signal section from an input signal containing at least the 
first signal section and the remaining signal section on a time division basis; 

extracting the signal of the first signal section out of the input signal in 
accordance with to the result of the detection of the first signal section; and 

recording the extracted first signal. 
[Claim 28] 

The signal processing method according to claim 27, further comprising the 
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steps of: 

extracting the characteristic values characterizing the first signal from the 
extracted first signal; and 

recording the characteristic values. 
[Claim 29] 

The signal processing method according to claim 27, wherein said first 
signal section detecting step is adapted to detect said first signal section from said 
input signal on the basis of a characteristic pattern of the signal appearing in said 
input signal at predetermined time intervals and a characteristic value showing the 
probability of the first signal. 
[Claim 30] 

The signal processing method according to claim 27, wherein 

said first signal section detecting step is adapted to detect said first signal 

section on a basis of predetermined guide information which is prepared to 

correspond to said input signal. 
[Claim 31] 

The signal processing method according to claim 28, wherein said 
characteristic value characterizing said first signal is amplitude of the signal in the 
first signal section. 
[Claim 32] 
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The signal processing method according to claim 28, wherein said 
characteristic value characterizing said first signal is a spectrum of the signal in the 
first signal section. 
[Claim 33] 

The signal processing method according to claim 28, wherein said 
characteristic value characterizing said first signal is a linear prediction coefficient 
of the signal in the first signal section. 
[Claim 34] 

The signal processing method according to claim 28, wherein said 
characteristic value characterizing said first signal is a histogram of a 
predetermined component of the signal in the first signal section. 
[Claim 35] 

The signal processing method according to claim 28, wherein said 
characteristic value characterizing said first signal is mean value of the 
predetermined component of the signal in the first signal section. 
[Claim 36] 

The signal processing method according to claim 28, wherein said 
characteristic value characterizing said first signal is difference of the 
predetermined signal component of the signal in the first signal section. 
[Claim 37] 
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The signal processing method according to claim 28, wherein said 
characteristic value characterizing said first signal is the number of changes of the 
state of the signal in the first signal section. 
[Claim 38] 

The signal processing method according to claim 28, wherein said 
characteristic value characterizing said first signal is the time of the change of the 
state of the signal in the first signal section. 
[Claim 39] 

The signal processing method according to claim 27, further comprising the 
steps of: 

extracting information to be used as an index representing said recorded 
first signal; and 

displaying said extracted index information. 
[Claim 40] 

The signal processing method according to claim 39, wherein said index 
information is an edited signal obtained by editing said first signal. 
[Claim 41] 

The signal processing method according to claim 40, wherein said edited 
signal obtained by editing said first signal comprises a set of signals at the time 
when the state of said first signal changes. 
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[Claim 42] 

The signal processing method according to claim 40, wherein said edited 
signal obtained by editing said first signal comprises a signal representing a starting 
part of said first signal and/or a signal representing an ending part of said first 
signal. 
[Claim 43] 

The signal processing method according to claim 39, wherein said index 
information extracting step is adapted to extract the signal at a time when the state 
of said first signal changes. 
[Claim 44] 

The signal processing method according to claim 28, further comprising a 
comparing step for comparing characteristic values respectively characterizing 
different first signals. 
[Claim 45] 

The signal processing method according to claim 44, wherein said 
comparing step is adapted to detect the agreement/disagreement of the first signals 
in a part of the section or in the entire section by comparing said characteristic 
values. 
[Claim 46] 

The signal processing method according to claim 44, wherein said 
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comparing step is adapted to detect the degree of similarity of the first signals in a 
part of the section or in the entire section by comparing said characteristic values. 
[Claim 47] 

The signal processing method according to claim 44, wherein said 
comparing steps is adapted to perform the comparing operation on the basis of the 
distance as determined by using a predetermined distance scale between vectors 
formed at least one of the amplitude of the signal in the first signal section, the 
spectrum of the signal in the first signal section, the linear prediction coefficient of 
the signal in the first signal section, the histogram of a predetermined component of 
the signal in the first signal section, the average value of a predetermined 
component of the signal in the first signal section, the difference in a predetermined 
signal component of the signal in the first signal section, the number of changes in 
the state of the signal in the first signal section and the time of a change in the state 
of the signal in the first signal section. 
[Claim 48] 

The signal processing method according to claim 39, further comprising an 
index information specifying step for specifying desired index information from 
said displayed plurality of pieces of index information; and 

a retrieving step for retrieving the first signal corresponding to said 
specified index information. 
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[Claim 49] 

The signal processing method according to claim 28, further comprising a 
retrieving step for retrieving a first signal substantially agreeing with said first 
signal from said recording step, using said first signal in a part of the section or in 
the entire section or a characteristic value characterizing the first signal as 
retrieving condition. 
[Claim 50] 

The signal processing method according to claim 28, further comprising a 
retrieving step for retrieving a first signal substantially agreeing with said first 
signal from said recording step, using a part or all of said first signal or a 
characteristic value characterizing the first signal as retrieving condition. 
[Claim 51] 

The signal processing method according to claim 27, further comprising a 
measuring step for measuring the number of times and/or the hours of appearances 
of a same first signal. 
[Claim 52] 

The signal processing method according to claim 27, further comprising a 
measuring step for measuring the number of times and/or the hours of appearances 
of similar first signals. 
[Claim 53] 
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The signal processing device according to claim 1, wherein said input 
signal comprises a video signal and/or an audio signal and said first signal covers a 
commercial message section. 
[Claim 54] 

The signal processing method according to claim 27, wherein said input 
signal comprises a video signal and/or an audio signal and said first signal covers a 
commercial message section. 
[Detailed Description of the Invention] 
[1] 

[Technical Field of the Invention] 

This invention is applied to an audio/video recording/reproduction 
apparatus. More particularly, the present invention relates to a signal processing 
device and a signal processing method that can be conveniently used for storing, 
viewing/listening to and/or retrieving commercial messages added to a television 
broadcast program. 
[2] 

[Prior Art] 

Conventionally, television and radio programs are normally broadcast with 
commercial messages (to be referred to simply as CMs hereinafter) provided by 
one or more than one commercial companies and/or organizations and inserted into 
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the program with few exceptions that may charge a subscription fee. 
[3] 

Some of the viewers of such a program may feel that the CMs are not 
wanted while other may appreciate them as necessary and direct sources of 
commercial information particularly when they present famous actors and/or 
interesting scenes. 
[4] 

However, while the channels and the time slots of television and radio 
programs to be broadcast can be known in advance by referring to a published time 
schedule, it is not easy to know in advance the channel and the time slot of a 
particular CM. Therefore, it may be highly convenient to those viewers/listeners 
who want to view/listen to particular CMs if the CMs that are broadcast and/or to 
be broadcast are stored at a particular site so that any viewer/listener who wants to 
view/listen to a particular CM may access, retrieve and view/listen to it at any time 
he or she wants. 
[5] 

[Problems to be Solved by the Invention] 

Japanese Patent Applications Laid-Open Publication Nos. 8-317342, 
3-158086 and 3-2622872 proposes techniques for detecting CMs out of broadcast 
signals so that a viewer/listener can view/listen to a program by skipping the CMs 
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in the program. However, none of the above cited patent documents describe a 

technique for storing accessing, retrieving and viewing/listening to CMs. 

[6] 

Meanwhile, Japanese Patent Application Laid-Open Publication No. 
10-224724 describes a technique of detecting information accompanying CMs, 
storing the information and replaying a same CM for once. However, the 
disclosed technique cannot be used to store, access, retrieve and view/listen to CMs. 
Additionally, the use of the disclosed technique is very limited because it cannot 
discriminate a stereophonic program and a CM that is broadcast in a stereo mode, 
although CMs are often broadcast in a stereo mode. 
[7] 

Therefore, there is a strong demand for a technique that can accurately 
detect CMs out of broadcast signals and also be used for storing, accessing, 
retrieving and viewing/listening to CMs. 
[8] 

In view of the above identified circumstances, it is therefore the object of 
the present invention to provide a signal processing device and a signal processing 
method that can accurately detect CMs out of broadcast signals and also be used for 
storing, accessing, retrieving and viewing/listening to CMs. 
[9] 
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[Means to Solve the Problem] 

According to the invention, the above object is achieved by providing a 
signal processing device comprising a first signal section detecting means for 
detecting a first signal section out of an input signal including at least the first 
signal section and the remaining signal section on a time division basis, a first 
signal extracting means for extracting the signal of the first signal section out of the 
input signal according to the result of the detection of the first signal section and a 
recording means for recording the extracted first signal. 
[10] 

In another aspect of the invention, there is also provided a signal processing 
method comprising steps of detecting a first signal section out of an input signal 
containing at least the first signal section and the remaining signal section on a time 
division basis, extracting the signal of the first signal section out of the input signal 
according to the result of the detection of the first signal section and recording the 
extracted first signal. 
[11] 

Thus, according to the invention, the first signal is detected out of the input 
signal and recorded and therefore, it is now possible to separate and sort out the 
first signal from the remaining signal section of the input signal. Additionally, 
according to the invention, it is easy to view, remove and/or detect the first signal 
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because some or all of the characteristic values characterizing the detected first 
signal can be recorded with the first signal itself. Still additionally, according to 
the invention, it is possible to extract and display the index information 
representing the first signal so that the first signal can be viewed and retrieved with 
ease. Furthermore, according to the invention, any duplicative parts of the 
recorded first signal can be removed to improve the recording efficiency and 
eliminate the possible waste of time of viewing a same signal for a number of times. 
Finally, according to the invention, the first signal can be used to detect a signal 
that is partly identical with or similar to the first signal. 
[12] 

[Preferred Embodiment of the Invention] 

Now, the present invention will be described in greater detail by referring to 
the accompanying drawings that illustrate a preferred embodiment of the invention. 
[13] 

FIG. 1 is a schematic block diagram of a preferred embodiment of 
broadcast signal processing device according to the invention. 
[14] 

Referring to FIG. 1, the broadcast signal processing device is adapted to 
receive an input signal 200a that can contain a video signal, an audio signal, a 
control signal and a program guide signal. The input signal 200a may be a signal 
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that is received by way of a ground wave broadcasting system, a satellite 
broadcasting system, a cable broadcasting system or a telephone network or 
reproduced from some other recording device that has recorded the signal in 
advance. The video signal and the audio signal take a major part of the broadcast 
signal and contain the signals of a program itself and one or more than one CMs. 
The control signal includes information on the broadcasting, the time slot of 
broadcasting the program, the frequency of the wave or the channel to be used for 
the transmission of the signal. The program guide signal contains data related to 
the video signal and the audio signal when the signals are received in the case of 
digital broadcasting or by way of a telephone network. While it is assumed that 
the signals are those of a single broadcast program, the present invention is also 
applicable to signals that are input simultaneously by way of a plurality of 
channels. 
[15] 

Of the broadcast signal processing device shown in FIG. 1, the CM 
detecting section 202 detects at least the start time of each CM (to be referred to as 
CM start time hereinafter) and the length of the CM section (to be referred to as 
CM length hereinafter from the input signal 200a containing the video signal, audio 
signal, control signal and the program guide signal and outputs a CM detection 
signal 202a indicating the CM start times and the CM lengths. The operation of 
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the CM detecting section 202 will be described in greater detail hereinafter along 
with the CM start times and the CM lengths. The CM detection signal 202a 
detected by the CM detecting section 202 is then sent to the CM extracting section 
201 and the CM characteristics extracting section. 
[16] 

The CM extracting section 201 extracts signal 201a that corresponds to the 
CM sections out of the input signal 202a on the basis of the CM detection signal 
202a supplied from the CM detecting section 202. More specifically, the CM 
extracting section 201 extracts the signal sections corresponding to the CM sections 
as defined respectively by the CM start times and the CM lengths indicated by the 
CM detection signal 202a fed from the CM detecting section 202 and outputs CM 
part signals corresponding to the respective CM sections and containing video 
signals, audio signals, control signals and program guide signals respectively. 
The CM part signals 201a are sent to the CM recording section 205. Since the 
CM detection signal 202a from the CM detecting section 202 is delayed relative to 
the input signal 200a by the time period required for the detecting operation, the 
CM extracting section 201 is adapted to absorb the delay by using a temporary 
storage such as a magnetic recording device that is internally or externally 
provided. 
[17] 
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The CM recording section 205 is in fact a device for recording and 
reproducing signals by using a magnetic tape, a magnetic disc, an opto-magnetic 
disc, a recordable optical disc and/or a semiconductor memory. Upon receiving 
the CM part signals 201a (including the video signals, the audio signals, the control 
signals and the program guide signals corresponding to the CM sections), the CM 
recording section 205 records the CM part signals 201a. 
[18] 

FIG. 2 is a schematic block diagram of a subset of the embodiment 
including only the CM extracting section 201, the CM recording section 202 and 
the CM recording section 205 and FIG. 3 is a flow chart of the operation of the 
subset of FIG. 2. 
[19] 

Referring to FIG. 3, as the video signals, the audio signals, the control 
signals and the program guide signals of the input signal 200a are sequentially 
input to the subset in step S220, the input signal 200a is stored in the internal or 
external temporary storage by the CM extracting section 201 in step S221. 
[20] 

At the same time, the CM detecting section 202 of the above subset detects 
the CM start times and the CM lengths of the CM sections. More specifically, the 
CM detecting section 202 detects each CM section out of the input signal 200a in 
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Step S222 and then processing operation proceeds to Step S223 when it is 
determined that the detected section is a CM section but to Step S225 when it is 
determined that the detected section is not a CM section. 
[21] 

In Step S223, the CM extracting section 201 extracts the signal 201a 
containing the video signal, the audio signal, the control signal and the program 
guide signal of the detected CM section. Then, in Step S224, the CM recording 
section 205 stores the extracted signal. 
[22] 

When the processing operation proceeds to Step S225, the signals 
temporarily stored by the CM extracting section 201 is discarded and the 
processing operation returns to Step S220 to repeat the above steps. 
[23] 

Referring back to FIG. 1, the CM characteristics extracting section 203 
extracts CM characteristics out of the input signal 200a on the basis of the CM 
detection signal 202a fed from the CM detecting section 202. More specifically, 
the CM characteristics extracting section 203 extracts the characteristic values 
(representing some or all of the characteristics as CM as will be described 
hereinafter) of each of the signal parts contained in the CM detection signal 202a 
fed from the CM detecting section 202 and corresponding to the CM sections as 
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defined by the CM start times and the CM lengths and outputs the characteristic 
values 203a to the CM recording section 205. Note that, since the CM detection 
signal 202a fed from the CM detecting section 202 is delayed relative to the input 
signal 200a by the time period required for the detecting operation, the CM 
characteristics extracting section 203 is adapted to absorb the delay by using a 
temporary storage such as a magnetic recording device that is internally or 
externally provided. The temporary storage of the CM characteristics extracting 
section 203 may share the temporary storage of the CM extracting section 201. 
[24] 

The characteristic values for characterizing each of the CMs are extracted 
from the video signal and the audio signal. Part or all of the physical values of 
each CM including the amplitude characterizing value, the spectrum characterizing 
value, the linear prediction coefficient, the brightness histogram, the color 
histogram, the average luminance the luminance difference energy, the number of 
cut changes and the number of each cut change may be used for the purpose of the 
invention. These characteristic values shows a pattern that is identical to a CM 
and but different from that of some other CM so that they can be used to 
characterize the CM. 
[25] 

In the following description, it is assumed that the input audio signal and 
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the input video signal are discretized and the input audio signal is expressed by 
S[m] as shown in (b) of FIG. 4, where m = 0, 1, M-l, representing discretized 
time periods, M being the discrete time corresponding to the frame being processed. 
Also assume that the input video signal is expressed by I[x, y; 1], where 1 = 0, 1, 
L-l, representing the video fame of the input image, L being the number of video 
frames corresponding to the processing frame, x = 0, X-l, representing the pixel 
number in the horizontal direction, X being the horizontal image size, and y = 0, 
X-l, representing the pixel number in the vertical direction, Y being the vertical 
image size. Note that the processing frame is the processing unit having a 
predetermined time length, which is typically 250ms. The number of frames 
corresponding to the length of a CM section is expressed by N as shown in (c) of 
FIG. 4. 
[26] 

Now, each characteristic value will be discussed below. 

[27] 

Firstly, the amplitude characteristic value A[n] of processing frame n is the 
value of the mean square of the audio signals in frame n and obtained by means of 
formula (1) below. 
[28] 

[Equation 1] 
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A[n) = — Y,S 2 [m + nM ] (1) 



[29] 



The vector of the amplitude characteristic value obtained for all the frames 
of the CM section is expressed by A = (A[0], A[l], A[N-1]). 
[30] 

The spectrum characteristic value F[k; n] of processing frame n is the 
average spectrum in frame n and obtained by means of formula (2) below; 
[31] 



[Equation 2] 



F[k;n] 



(2), 



[32] 



where k = 0, K, representing the discretized frequency, K being the maximum 
frequency to be treated. The computation is carried out by typically using FFT or 
linear prediction analysis. The operation of discretization of frequency can be 
reduced to a linear one or non-linear one that uses octave as unit of operation by 
rediscretizing k in formula (2). The vector of the spectrum characteristic value 
obtained for all the frames of the CM section is expressed by F = (F[0;0], F[1;0], 
F[K-1;N-1]). 
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[33] 

The linear prediction coefficient P[k;n] of processing frame n can be 
computationally determined by using an LPC algorithm such as the one shown in 
Linear Prediction of Speech (Markel et al., Springer Verag, 1978). k = 0, 
K-l denotes the number for the linear prediction coefficient and K denotes the 
degree of prediction. The vector of the linear prediction coefficient obtained for 
all the frames of the CM section is expressed by P = (P[0;0], P[1;0], 
P[K-1;N-1]). 
[34] 

The luminance histogram Hj[q;n] of processing frame n is the luminance 
histogram of the video signals of frame n. q = 0, Q-l denotes the index 
number of the luminance slot and Q denotes the number of slots of the histogram. 
[35] 

The color histogram H c [q;n] of processing frame n is the intensity 
histogram of each color of the signal of frame n. q = 0, Q-l denotes the index 
number of the color and intensity slot and Q denotes the number of slots of the 
histogram. 
[36] 

The vector of the luminance histogram and that of the color histogram 
formed for all the frames of the CM section are expressed respectively by Hj = 
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(H,[0;0], H,[1;0], H^Q-ljN-l]) andH c = (H c [0;0], H C [1;0], H C [Q-1;N-1]). 
[37] 

The average luminance B[n] of processing frame n is that of the video 
signals in processing frame n and determined by formula (3) below. 
[38] 

[Equation 3] 

«W~III/[x,,;/] 

^ 1Ll /=0 jc=0 y=0 

[39] 

The vector of the average luminance B[n] obtained for all the frames of the 
CM section is expressed by B = (B[0], B[l], B[N-1]). 
[40] 

The luminance difference energy D[n] of processing frame n is the 
inter-pixel difference energy of adjacent video frames and typically determined by 
formula (4) below. 
[41] 

[Equation 4] 

Din)= Y J T n LLI(/[x,y;/]-/ky;/-l]) 

[42] 
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The vector of the luminance difference energy obtained for ail the frames of 
the CM section is expressed by D = (D[0], D[l], D[N-1]). The inter-pixel 
difference may be replaced by the difference in the average luminance of all the 
images or the inter-block difference, using blocks of 8 x 8 pixels or 16 x 16 pixels 
for horizontal direction x vertical direction. 
[43] 

The number of cut changes C[n] of processing frame n is the number of the 
frames where the pixel difference energy between adjacent video frames exceeds a 
predetermined threshold value in the processing frame n and determined by 
formula (5) below; 
[44] 

[Equation 5] 

LL(/[*,y;/]-/[*,y;/-iD *D t 

[45] 

where Count b a (f) is a function for showing the number of times where the 
requirement of evaluation formula f is satisfied and Dthsd is a predetermined 
threshold value for the luminance difference energy. The vector of the number of 
cut changes obtained for all the frames of the CM section is expressed by C = (C[0], 
C[l], ...,C[N-1]). 
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[46] 

The time of each cut change can be determined in terms of frame number n 
that satisfies the requirement of C[n] > 0. The number of cut changes for all the 
CM section can be determined as the total sum of C[n]. 
[47] 

The CM characteristics extracting section 203 detects all or part of the 
above described characteristic values for each CM section. Therefore, the 
characteristic values that characterizes the CM section is expressed by vector V 
defined by formula (6) below. 
[48] 

V = (A, F, P, Hi, He, B, D, C) (6) 

The characteristic vector V can also be defined by formula (7) below; 

[49] 

V = (V[0],V[1],...,V[N-1]) (7) 

where V[n] is the vector of the characteristic values of processing frame n and 

defined by formula (8) below. 

[50] 

V = (A[n], F[k;n], P[k;n], H^n], H c [n], B[n], D[n], C[n]) (8) 

The characteristic values described above and extracted by the CM 
characteristics extracting section 203 are then stored in the CM recording section 
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205 with the signal of the CM section (including video signal/audio signal/control 
signal/program guide signal) for the CM section extracted by the CM extracting 
section 201, which is described earlier. 
[51] 

FIG. 5 is a schematic block diagram of a subset of the embodiment 
including the CM extracting section 201, the CM detecting section 202, the CM 
characteristics extracting section 203 and the CM recording section 205 and FIG. 6 
is a flow chart of the operation of the subset of FIG. 5. Note that the steps of the 
flow chart of FIG. 6 that are same as those of the flow chart of FIG. 3 are denoted 
respectively by the same step numbers and will not be described any further. 
Therefore, only the steps relating to the CM characteristics extracting section 203 
will be described below. 
[52] 

Referring to FIG. 6 illustrating the operation of the subset of FIG. 5, after 
Step S220, where the video signal, the audio signal, the control signal and the 
program guide signal of the input signal 200a are sequentially input, the processing 
operation proceeds to Step S23 1 , where the input signal 200a is temporarily stored 
in the internal or external temporary storage of the CM characteristics extracting 
section 203. 
[53] 



30 



At the same time, the start time of the CM section and the time length of the 
CM section are detected by the CM detecting section 202 and the processing 
operation proceeds to Step S233 if it is determined in Step S222 that the detected 
section is a CM section, whereas the processing operation proceeds to Step S235 if 
it is determined in Step S222 that the detected section is not a CM section. 
[54] 

As for the subset of FIG. 5, as the processing operation proceeds to Step 
S232, the CM characteristics extracting section 203 extracts characteristic values of 
the CM section out of the video signal, the audio signal, the control signal and the 
program guide signal of the CM section and the extracted characteristic values are 
stored in the CM recording section 205 in Step S234. 
[55] 

Then, the processing operation proceeds to Step S235, where the signals 
temporarily stored in the CM characteristics extracting section 203 are discarded, 
and returns to Step S220 to repeats the above steps. 
[56] 

Returning to FIG. 1, the CM part signals 201a (including the video signals, 
the audio signals, the control signals and the program guide signals) of the CM 
section stored in the CM recording section 205 and the characteristic values 203a of 
the CM section are read out of the CM recording section 205 and sent to the CM 
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index generating section 206 and the CM viewing section 208. 
[57] 

The CM index generating section 206 generates information to be used as 
indexes for the CM on the basis of the CM part signals 201a and the characteristic 
values 203a fed from the CM recording section 205 and sends the information 206a 
(to be referred to as CM index information hereinafter) to the CM viewing section 
208. 
[58] 

The CM index information 206 a may typically be information showing the 
starting image, the cut point image, the cut point video image, the starting sound 
and/or the ending sound of the CM, which will be described below. 
[59] 

Now, the CM index information 206a will be discussed below. 

[60] 

Firstly, the starting point image is the image at the very start of the CM 
section and used as part of the index information. 
[61] 

The cut point image is the image of a cut point of the CM section and used 
as part of the index information. Since a cut point belongs to a processing frame 
whose characteristic value C[n] is greater than 0, the image of the time is used. 
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While a CM section normally has a plurality of cut points, only a single image that 
may be the image of the first cut point or that of the last cut point as selected 
according to a predetermined criterion is used as the cut point image. 
[62] 

The cut point video image is obtained by connecting the images of the cut 
points of the CM section to produce a video image, which is then used as part of 
the index information. Since a CM section normally has a plurality of cut points, 
a short video image is generated by connecting all the images of all the cut points. 
[63] 

The starting sound is a sound that lasts for a predetermined period of time, e. 
g., 2 seconds, from the very start of the CM section and used as part of the index 
information. Since CMs often have a short characteristic sound at the very start 
thereof, it is useful to utilize the starting sound as a CM index. 
[64] 

The ending sound is a sound that lasts for a predetermined period of time, e. 
g., 2 seconds, until the very end of the CM section and used as part of the index 
information. Since CMs often have a sound and an image specific to a particular 
product or a particular company or some other organization, it is useful to utilize 
the ending sound as a CM index. 
[65] 
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The CM part signals 201a and the characteristic values 203a of the CM 
section from the CM recording section 205 (to be collectively referred to as 
recording section reproduced signal 205a whenever appropriate hereinafter) are 
supplied to the CM viewing section 208 along with the CM index information 206a 
from the CM index generating section 206. The CM viewing section 208 includes 
a display processor 801 and a display section 802. 
[66] 

The display section 802 typically comprises a display device such as a CRT 
(cathode ray tube) or a liquid crystal monitor and a loudspeaker and is adapted to 
provide the user with images and sounds. 
[67] 

The display processor 801 of the CM viewing section 208 is also fed with 
the user selected information 207a generated by the CM selecting section 207 
according to the selecting instruction 209a of the user 209. As will be described 
hereinafter, the user 209 will view (208a) the CM index images, or the images and 
the icons, displayed on the display section 802 and issues a selecting instruction 
209a to the CM index images, or the images and the icons, displayed on the display 
section 802 typically by means of a pointing device which may be a mouse, a 
remote commander or a touch panel. Then, the CM selecting section 207 
generates user selection information 207a corresponding to the user's selecting 
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instruction 209a. The generated user selection information 207a is sent to the 

display processor 801 of the CM viewing section 208. 

[68] 

The display processor 801 receives the CM index information 206a and the 
recording section reproduced signal 205a (particularly, video/audio signal sections) 
along with the user selection information 207a from the CM selecting section 207 
and operates to display information to the user. The display processor 801 may 
typically comprise a processor and software. 
[69] 

Now, the operation of the display processor 801 will be discussed below by 
referring to FIG. 7 showing a typical image of the display section 802. 
[70] 

The display processor 801 firstly arranges CM index images, or video 
images 810, on a plurality of CMs according to the CM index information 206a 
typically as a single scene and displays it on the display section 802 as shown in 
FIG. 7. Each of the CM index images, or the video images 810, may be the 
starting point image or the cut point video image of the corresponding CM. If the 
number of CMs (the CM index images or the video images 810) is large, they may 
be arranged in two or more than two scenes. The display processor 801 also 
arranges icons 811 through 814 to be used by the user to input his or her 
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instructions to the CM selecting section 207 in a scene along with the CM index 
images or the video images 810 on the display section 802. In the instance of FIG. 
7, a total of twelve CM index images or the video images 810 are displayed in a 
single scene along with a CM reproduction icon 811, a sound reproduction icon 
812, a preceding page icon 813 and a succeeding page icon 814. 
[71] 

When a scene as shown in FIG. 7 is displayed and the display processor 
801 receives user selection information 207a from the CM selecting section 207, 
the display processor 801 changes the display according to the user selection 
information 207a. More specifically, if the user 209 selects one or more than one 
CM index images or the video images by way of the CM selecting section 207, the 
display processor 801 may highlight the selected CM index images or the video 
images 810. 
[72] 

Additionally, when a scene as shown in FIG. 7 is displayed and the user 
209 selects one of the icons by way of the CM selecting section 207, the display 
processor 801 carries out a processing operation that corresponds to the selected 
icon. If, for example, the CM reproduction icon 811 is selected by the user 
selection information 207a, the display processor 801 reproduces the image and the 
sound of the CM that is already selected from the CM index images or the video 
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images 810. If the sound reproduction icon 8 1 2 is specifically selected by the user 
selection information 207a, the display processor 801 reproduces the starting sound, 
the ending sound or the entire sound of the CM. If the preceding page icon 813 is 
selected by the user selection information 207a, the display processor 801 displays 
the CM index images or the video images 810 of the immediately preceding page 
on the display screen. If, on the other hand, the succeeding page icon 814 is 
selected by the user selection information 207a, the display processor 801 displays 
the CM index images or the video images 810 of the immediately succeeding page 
(that are not viewed by the user) on the display screen. 
[73] 

FIG. 8 is a schematic block diagram of a subset of the embodiment 
including the CM recording section 205, the CM index generating section 206, the 
CM viewing section 208 and the CM selecting section 207. FIG. 9 is a flow chart 
of the operation of the subset of FIG. 8, that of the CM viewing section 208 in 
particular. 
[74] 

Referring to FIG. 9, the CM viewing section 208 specifies the CMs of the 
first page in Step S240 and obtains the video signals/audio signals of the CMs of 
the first page in Step S241. Then, CM index information 206a is generated in 
Step S242 by the CM index generating section 206. 
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[75] 

Subsequently, in Step S243, the display processor 801 of the CM viewing 
section 208 generates a scene to be displayed on the display screen according to the 
CM index information 206a and drives the display section 802 to display the scene. 
Then, in Step S244, the display processor 801 waits for the input of a user 
instruction (user selection information 207a), while displaying the scene. 
[76] 

If the user selection information 207a generated by the CM selecting 
section 207 according to the input of the instruction of the user 209 specifies a CM 
index, the display processor 801 selects the CM index image or the video image 
810 specified by the user selecting information 207a in Step S245. Then, the 
display processor 801 regenerates the scene highlighting the CM index image or 
the video image 810 and causes the display section 802 to display the highlighted 
scene as in Step S243 and subsequently waits for the input of an instruction by the 
user 209 as in Step S244. 
[77] 

If the user selection information 207a specifies the CM reproduction icon 
811, the display processor 801 reproduces in the CM recording section 205, the 
video signal and the audio signal of the CM corresponding to the CM index image 
or the video image 810 already specified by the user selection information 207a, in 
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Step S246. Then, the display processor 801 regenerates the scene corresponding 
to the reproduced video signal and causes the display section 802 to display the 
scene as in Step S243 and subsequently waits for the input of an instruction by the 
user 209 as in Step S244. 
[78] 

If the user selection information 207a specifies the sound reproduction icon 
812, the display processor 801 reproduces in the CM recording section 205, the 
audio signal of the starting sound, the ending sound or all the sound of the CM 
corresponding to the CM index image or the video image 810 already specified by 
the user selection information 207a, in Step S247. Then, the display processor 
801 drives the loudspeaker to output the sound of the reproduced audio signal as in 
Step S243 and subsequently waits for the input of an instruction by the user 209 as 
in Step S244. 
[79] 

If the user selection information 207a specifies the preceding page icon 813, 
the display processor 801 reproduces in the CM recording section 205, the video 
signals and the audio signals of the CMs corresponding to the CM index images or 
the video images 810 of the immediately preceding page, in Step S248. Then, the 
display processor 801 drives the display section 802 to display the images 
corresponding to the reproduced video signals as in Step S243 and subsequently 
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waits for the input of an instruction by the user 209 as in Step S244. 
[80] 

Finally, if the user selection information 207a specifies the succeeding 
page icon 814, the display processor 801 reproduces in the CM recording section 
205, the video signals and the audio signals of the CMs corresponding to the CM 
index images or the video images 810 of the immediately succeeding page, in Step 
S249. Then, the display processor 801 drives the display section 802 to display 
the images corresponding to the reproduced video signals as in Step S243 and 
subsequently waits for the input of an instruction by the user 209 as in Step S244. 
[81] 

The above steps will then be repeated appropriately. 

[82] 

Returning to FIG. 1, the characteristic vector V of each CM recorded in the 
CM recording section 205 is sent to the characteristics comparing section 204 as 
signal 205b. 
[83] 

The characteristics comparing section 204 reads out the characteristic 
vector V of each CM recorded in the CM recording section 205 by means of a 
reading control signal 204a. The characteristics comparing section 204 
determines by comparison if the CM is the right CM or not by means of the 
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characteristic vector Vj of the CM read out from the CM recording section 205. 
Note that "i" denotes the index (the variable of index) discriminating the individual 
CMs. The characteristics comparing section 204 compares the characteristic 
vectors Vj and Vj of two CMs in a manner as shown below. 
[84] 

Firstly, it carries out computations for an evaluation function J(i,j), using 
formula (9) below. 
[85] 

[Equation 6] 

(9) 

[86] 

Subsequently, it compares J(i,j) with a predetermined threshold value J ths d 
and determines to be same if J(i,j) is lower than Jth sd and to be different if J(i,j) is 
not lower than J^sd- The characteristics comparing section 204 carries out the 
comparing operation for all the CMs stored in the CM recording section 205 and 
removes either one of the two CMs that are determined to be same from the CM 
recording section 205. As a result, all the duplicate signals of CMs stored in the 
CM recording section 205 will be removed. 
[87] 
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FIG. 10 is a schematic block diagram of a subset of the embodiment 
including the CM recording section 205 and the characteristics comparing section. 
FIG. 11 is a flow chart of the operation of the subset of FIG. 10, that of the 
characteristics comparing section 204 in particular. 
[88] 

Referring to FIG. 11, the characteristics comparing section 204 sets the 
variable i of said index to 0 in Step S250 and then sets the variable j of said index 
toi+1. 
[89] 

Subsequently, the characteristics comparing section 204 carries out 
computations for the evaluation function J(i,j) on the basis of the characteristic 
vectors V, and Vj and compares it with a predetermined threshold value J t h S d in Step 

5252. If the value of the function J(i,j) is lower than the threshold value Jthsd 
(Yes), the two CMs are determined to be same and the CM represented by the 
characteristic vector Vj is removed from the CM recording section 205 in Step 

5253. If, on the other hand, the value of the function J(i,j) is not lower than J t h S d 
(No), the two CMs are determined to be different from each other and the 
processing operation proceeds to Step S254. 

[90] 

As the processing operation proceeds to Step S254, the characteristics 
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comparing section 204 checks if the value of j corresponds to the last CM number 
among the CMs to be compared or not. If it is determined in Step S254 that the 
value of j does not correspond to the last CM member (No), j is incremented in 
Step S255 and the processing operation returns to Step S252, where J(ij) is 
compared with the threshold value once again. If, on the other hand, it is 
determined in Step S254 that the value of j corresponds to that last CM member 
(Yes), it is determined in Step S256 if i corresponds to the last CM number or not. 
If it is determined in Step S256 that the value of i does not correspond to the last 
CM number (No), i is incremented in Step S257 and the processing operation 
returns to Step S25 1 to set j once again. If, on the other hand, it is determined in 
Step S256 that the value of i corresponds to the last CM number (Yes), the 
processing operation is terminated there. 
[91] 

In this embodiment of broadcast signal processing device, additionally 
retrieval icons may be added to the displayed scene of the display section 802 
illustrated in FIG. 7 so that the CM wanted by the user may be retrieved by 
specifying the corresponding retrieval icon. FIG. 12 schematically illustrates a 
scene displayed on the display section 802 and containing an additional retrieval 
icon 815. The operations of the CM recording section 205, the characteristics 
comparing section 204, the CM viewing section 208, the CM index generating 
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section 206 and the CM selecting section 207 will be described by referring to the 

illustrated image of FIG. 12. 

[92] 

As the retrieval icon 815 is specified by the user 209, the CM selecting 
section 207 generates user selection information 207a corresponding to the user's 
selection and transmits the user selection information 207a to both the CM viewing 
section 208 and the characteristics comparing section 204. 
[93] 

Upon receiving the user selection information 207a, the characteristics 
comparing section 204 retrieves the CM having characteristics whose values 
partially agree with those of the CM that is already selected in the CM index image 
or the video image 810. 
[94] 

More specifically, when a CMi is selected by the CM selecting section 207 
according to the user selection information 207a, the characteristics comparing 
section 204 compares the characteristic values of the CMi with the characteristic 
values of the CMj that is recorded in the CM recording section 205. 
[95] 

For carrying out this comparing operation, the characteristics comparing 
section 204 firstly carries out computations for the evaluation function J'(ij) of the 
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CM part section, using formula (10) below; 
[96] 

[Equation 7] 

(10) 

[97] 

where N s denotes the first processing frame number of the CM part section 
to be compared and N e denotes the last processing frame number of the CM part 
section to be compared. 
[98] 

Subsequently, the characteristics comparing section 204 compares the 
evaluation function J'(iJ) with the predetermined threshold value J' thsd and 
determines that they do not agree with each other when the former is greater than 
the latter value but they do when the former is not greater than the latter. 
[99] 

It should be noted here that different CMs provided by a same sponsor may 
often be so designed as to make the viewer have a feeling of sameness or similarity 
by using common video/audio signals that are specific to a particular product or a 
particular company or some other organization. Then, the common video/audio 
signals will normally last for about a second. Therefore, if N e = N-l is used for 
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the last frame number of the CM section and N s is used for the processing frame 
preceding the last frame by one second, all the CMs provided by a same company 
can be retrieved by specifying one of the CMs of the company. 
[100] 

The evaluation function J'(iJ) to be used for comparing characteristics 
values may alternatively be defined by formula (11) below; 
[101] 

[Equation 8] 

r{ij)=Min s X I^M-^Ml 

n " s " (11) 

[102] 

where N w denotes the span to be used for retrieving agreeing parts whose 
characteristic values agree with each other and s denotes the index number that is 
used for shifting the span for retrieving partially agreeing parts from the beginning 
to the end of the CM section. Min s () denotes the smallest value of all the values 
of s. It will be effective to use only the sound-related characteristic values A[n], 
F[k;n] and P[k;n] excluding the image-related characteristic values Hj[n], H c [n], 
B[n], D[n] and C[n] from the vector [n]. 
[103] 

By using such a function, it is possible to retrieve CMs that partly agree 
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with each other. For example, CMs containing a same piece of background music 
may be retrieved. If the CMs of a particular product or a particular company or 
some other organization are designed to provide a sense of continuation, they are 
often made to contain a same piece of background music in order to make the 
viewers have a feeling of sameness of similarity. Therefore, the function can be 
an effective evaluation function to be used for retrieving CMs that are designed to 
provide a sense of continuation. 
[104] 

The evaluation functions J(i,j) and J'(i,j) may be referred to as similarity of 
two CMs. 
[105] 

The audio and video signals of the CM retrieved from the CM recording 
section 205 are then sent to the CM viewing section 208 as recording section 
reproduced signal 205a. 
[106] 

At the same time, the index generating section 206 generates index 
information on the CMs whose characteristic values agree with each other and 
transmits the generated index information 206a to the CM viewing section 208. 
[107] 

As a result, the CM viewing section 208 can display the image of the 
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retrieved CM and/or reproduce the sound of the CM. 
[108] 

FIG. 13 is a schematic block diagram of a subset of the embodiment 
including the CM recording section 205, the characteristics comparing section 204, 
the CM index generating section 206, the CM viewing section 208 and the CM 
selecting section 207. FIG. 14 is a flow chart of the operation of the subset of FIG. 
13. It will be appreciated that the flow chart of FIG. 14 comes immediately after 
the flow chart of FIG. 9. 
[109] 

Referring to FIG. 14, when the processing operation waits for a user's 
instruction in Step S244 in the flow chart of FIG. 9 and the user selection 
information 207a specifies the CM retrieval icon 815, the characteristic comparing 
section 204 sets the index variable i to the value of the index of the selected CM in 
Step S260 and then sets the index variable j to 0 in Step S26 1 . 
[110] 

Subsequently, the characteristics comparing section 204 carries out 
computations for the evaluation function J'(ij) in Step S262, using the 
characteristic vectors Vj and Vj, and compares the obtained value with the 
predetermined threshold value J\h S d- If it is determined that the obtained value is 
not greater than the threshold value J' thsd (Yes), the CM is judged to be a similar 
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CM and the CM represented by the characteristic vector V) is marked in Step S263. 
If, on the other hand, it is determined that the obtained value is greater than the 
threshold value J' thsd (No), the CM is judged to be not a similar CM and the 
processing operation proceeds to the next step, or Step S294. 
[Ill] 

In Step S294, the characteristics comparing section 204 checks if the value 
of j corresponds to the last CM number among the CMs to be compared or not. If 
it is determined in Step S294 that the value of j does not correspond to the last CM 
member (No), j is incremented in Step S265 and the processing operation returns to 
Step S262, where J(i,j) is compared with the threshold value once again. If, on the 
other hand, it is determined in Step S266 that the value of j corresponds to that last 
CM member (Yes), all the marked CMs are specified collectively and the 
processing operation returns to Step S241 shown in FIG. 9, where signals of the 
specified CMs are retrieved from the CM recording section 205. 
[112] 

This embodiment may alternatively be so arranged that the same CMs are 
not removed but stored and, the number of times and the clock times when same 
CMs are broadcast by retrieving them, detecting the agreement in all the CM 
sections. 
[113] 
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As will be appreciated from the above description, with this embodiment of 
broadcast signal processing device, only the CM sections can be extracted from a 
broadcast signal and stored. Therefore, it is possible to provide a data base device 
adapted to store only the CM sections of a program without recording the program 
itself. 
[114] 

Additionally, with this embodiment of broadcast signal processing device, 
only the CM sections can be extracted from a broadcast signal along with 
characteristic values characterizing the CM sections so that the characteristic values 
may be stored. Then, it is possible to provide a data base device adapted to store 
only the CM sections of a program and their characteristic values without recording 
the program itself. Such characteristic values are useful when retrieving same 
CMs and/or similar CMs. 
[115] 

Still additionally, with this embodiment of broadcast signal processing 
device, any of the stored CMs can be displayed and viewed so that the viewer 
(user) can display a list of the stored CMs and reproduce and retrieve any CMs he 
or she wants. This functional feature is particularly useful when the viewer wants 
to detect and view a specific CM. 
[116] 
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Still additionally, with this embodiment of broadcast signal processing 
device, any duplicate CMs can be removed from the stored CMs to save the storage 
capacity and eliminate the trouble of repeatedly viewing same CMs. 
[117] 

Furthermore, with this embodiment of broadcast signal processing device, it 
is possible to retrieve similar CMs so that the CMs of a same product and the CMs 
of a same provider can be retrieved and displayed with ease. 
[118] 

Furthermore with this embodiment of broadcast signal processing device, it 
is possible to retrieve CMs having a same end part. Because CMs of a same 
product or a same company or some other organization normally contains a 
common image and a common sound, it is now possible to retrieve the CMs of a 
same product or a same provider can be retrieved and displayed with ease. 
[119] 

Still furthermore, with this embodiment of broadcast signal processing 
device, it is now possible to retrieve CMs containing a common piece of 
background music. Because CMs of a same product or a same company or some 
other organization often contains a common piece of background music, different 
CMs prepared with a sense of continuity can be retrieved and displayed without 
difficulty. 
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[120] 

Finally, with this embodiment of broadcast signal processing device, it is 
possible to observe the number of times and the clock times when same CMs are 
broadcast. This functional feature can be of great help to the producer of a CM 
when he or she compares each scheduled transmission of the CM and the 
corresponding actual transmission of the CM. 
[121] 

Now, the operation of the CM detecting section 202 of FIG. 1 for detecting 
CMs out of a broadcast signal along with the start time and the length of each CM 
will be discussed below in greater detail. 
[122] 

Firstly, the principle underlying the operation of detecting a CM section out 
of a TV broadcast signal of the CM detecting section 202 will be briefly described. 
In the following description, while the term "a characteristic value" may or may not 
mean same as the term "characteristic" as used above from viewpoint of the CM 
characteristics extracting section 203 and the characteristics comparing section 204, 
although the both are used to express a characteristic aspect of a CM. Therefore, 
the two terms are used here indiscriminately. 
[123] 

Generally, CMs that are broadcast through TV channels are prepared 
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according to the specifications provided by the broadcasters particularly in terms of 
duration and hence "the time length (of a CM) is limited and cannot vary in many 
different ways". For instance, in Japan, all CMs have a duration of 15 seconds, 30 
seconds or 60 seconds with very few exceptions. 
[124] 

Additionally, when a CM is broadcast, it is inserted into the stream of a TV 
program but prepared independently from the latter so that "the sound level falls 
(and becomes less audible)" and "the video signal is switched" inevitably before 
and after the CM. The expression that "the sound level falls" does not necessarily 
means that silence (or only a light noise) prevails there. In other words, the 
program can be switched to the CM or vice versa without a complete silence 
depending on the timing of the switch. 
[125] 

Almost all CMs show the three characteristic aspects of CMs including 
"specified time lengths (few variations in time length)", "a low sound level" and "a 
switched image". In other words, these are three "requirements" to be met by 
CMs. Therefore, these requirements will be referred to as "the three 
indispensable requirements" of CMs hereinafter. 
[126] 

Thus, it is deterministically possible to detect CM candidates (or signal 
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parts that will probably represents the corresponding respective CMs) practically 
without error by detecting the signal parts that meets the indispensable 
requirements out of a TV broadcast signal. However, since programs can also 
contain signal parts that do not represent any CM but meet the indispensable 
requirements, there still remains a risk of erroneously detecting parts of the 
program as CM candidates if only the indispensable requirements are used for 
detecting CMs. 
[127] 

On the other hand, many CMs show some or all of the following 
characteristic aspects due to of the nature of CMs, although there are obviously 
more exceptions if compared with the indispensable requirements. 
[128] 

1) The sound level can be more often than not falls immediately before and 
after a CM (immediately before the start of a CM and immediately after the 
end of the CM and before the restart of the program itself). 

[129] 

2) A silence of several hundred milliseconds can often appear between a CM 
and the program containing the CM or between a CM and another CM. 

[130] 

3) The sounded sections of a TV program normally have a length shorter than 
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the specified durations of CMs (15 seconds, 30 seconds, 60 seconds) by 
more than about hundred milliseconds and less than about a second. 

[131] 

4) The correlative value of the left channel (L-channel) and the right channel 
(R-channel) of a stereophonic sound signal is more often than not 
significantly smaller than 1 . 

[132] 

5) The sound level of CMs is often higher than that of programs. 
[133] 

6) CMs are often broadcast in a stereophonic mode 
[134] 

7) A plurality of CMs are often broadcast in a CM section. 
[135] 

8) More video cuts are used per unit time in CMs than in programs. 
[136] 

9) To the contrary, there are CMs containing extremely small video cuts per 
unit time (e.g., CMs using still images). 

[137] 

10) The sound quality can remarkably change at the boundary of a program and 
a CM or that of two CMs. 
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[138] 

11) CMs often contain both voices and music. 
[139] 

1 2) CMs are broadcast with a high probability on the hour. 
[140] 

13) Similarly, CMs are broadcast with a high probability on the half hour. 
[141] 

14) There are time slots where CMs are broadcast highly probably depending on 
the program category (e.g., during the half time of the live of a succor 
match) 

[142] 

For the purpose of the present invention, the requirements that arises from 
the above characteristic aspects of CMs will be referred to as "additional 
requirements". In other words, the additional requirements occurs from the fact 
that CMs are produced by conforming to given specifications, that CMs are 
intended to achieve an advertising effect in a very short period of time and that 
CMs are subject to programming restrictions. Therefore, while the additional 
requirements are not deterministically reliable, they are effective when evaluating 
the probability of each CM. 
[143] 
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In TV broadcasts, the images and sounds of two or more than two channels 
will never be transmitted through a single channel. In other words, if two video 
and audio sections that satisfy the above additional requirements are found in a TV 
broadcast signal and are overlapping each other and CM candidates are detected in 
such partly overlapping sections when detecting signal parts (CM candidates) from 
a TV broadcast signal, either of the overlapping video and sound sections cannot 
represent a right CM section. For the purpose of the invention, this requirement 
of TV broadcasts is referred to as "logical requirement". 
[144] 

Thus, for the purpose of the invention, the above described "indispensable 
requirements", "logical requirement" and "additional requirements" are rationally 
and effectively utilized to accurately detect CM sections from a TV broadcast 
signal. 
[145] 

More specifically, according to the invention, CM candidates (signal parts 
that probably represent respective CMs) are deterministically extracted from a TV 
broadcast signal on the basis of the "indispensable requirements" and selected 
according to statistic criteria that are based on the "additional requirements and any 
overlapping relationship of CMs is eliminated on the basis of the "logical 
requirement" to accurately detect CMs. 
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[146] 

FIG. 15 is a schematic block diagram of the CM detecting section, 
illustrating a possible first configuration thereof. Roughly speaking, the CM 
detecting section 202 comprises a front end section and a rear end section. In FIG. 
15, the operation control section 23 determines if the broadcasting channel being 
detected is a channel where CMs are broadcast or not on the basis of channel 
information lb that indicates the type of the channel and controls the components 
of FIG. 1 5 so as not to operate for detecting CMs if the channel is not used for 
broadcasting CMs. 
[147] 

Firstly, the front end section of FIG. 15 will be described. 

[148] 

Referring to FIG. 15, the video signal 2a, the audio signal 2b and the 
broadcasting mode signal 2c of the input signal 200a are input to the front end 
section. The time information 3 a generated by a clock (not shown) is also fed to 
the front end section. 
[149] 

The video signal 2a is digitized by A/D converter 10 and stored in frame 
memory 11, which is adapted to store at least video signals for two frames. The 
video signal read out from the frame memory 1 1 for each frame is then sent to cut 
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change detector 12. 
[150] 

The cut change detector 12 detects frames where the image changes 
abruptly (to be referred to as image change frames hereinafter) and frames showing 
a uniform luminance (to be referred to as uniform luminance frames hereinafter) on 
the basis of the video signal of each frame fed from the frame memory 1 1 . 
[151] 

More specifically, the cut change detector 12 determines the difference of 
the square sums of the pixels of the temporally adjacent frames stored in the frame 
memory 1 1 and detects the temporally succeeding one of the two adjacent frames 
as image change frame where the image changes abruptly if the difference of the 
square sums exceeds a predetermined threshold value. Additionally, the cut 
change detector 12 determines the variance of luminance of the image of each of 
the frames stored in the frame memory 11 and detects the frame as a uniform 
luminance frame if the variance of luminance of the frame is under a predetermined 
threshold value. If the time interval separating two adjacent frames (about 30ms 
in the case of the NTSC system) does not agree with the frame period in the audio 
signal processing operation which will be described hereinafter, it is made to agree 
with the frame period by rediscretizing the time interval of two adjacent frames. 
[152] 
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Now, the operation of detecting image change frames and uniform 
luminance frames of the cut change detector 12 will be described below in greater 
detail. 
[153] 

Assume that the transversal size and the longitudinal size of a discretized 
video signal are X and Y respectively, that the transversal and longitudinal pixel 
numbers of each pixel are expressed by x, y respectively and that the image of the 
n-th frame is expressed by I n (x,y) while the image of the (n-l)-th frame that 
temporally immediately precedes the n-th frame is expressed by I n _i(x,y). Then, 
the square sum D[n] of the difference of luminance of the pixels between the n-th 
frame and the (n-l)-th frame is obtained by formula (12) below and the variance of 
luminance of the n-th frame V[n] is expressed by formula (13) below. 
[154] 

[Equation 9] 



^W-LL(/.(^y)-Vi(*.y)) (12) 



1 x-iy-i i x-\y-\ 



(13) 



[155] 

The detection output C[n] of the cut change detector 12 is expressed by 
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formula (14) below; 
[156] 

[Equation 10] 

c ll(D[n]>D thsd orV[n]<V thsd ) 

{0(D[n]<D thsd andV[n]>V thsd ) ^ 

[157] 

where D thsd is the predetermined threshold value for the square sum that is used 
when detecting an image change frame and Vth sd is the predetermined threshold 
value for the variance of luminance that is used when detecting a uniform 
luminance frame. 
[158] 

The detection output C[n] of the cut change detector 12 is then sent to 
characteristic value buffer 1 8 as a characteristic value of the video signal. 
[159] 

When determining the difference of luminance between the two frame 
images, a memory having a storage capacity for storing video signals of two frames 
is required and the detector 12 is required to carry out arithmetic operations on the 
video signals of two frames. However, the storage capacity and the amount of 
arithmetic operations can be reduced by dividing a frame image into small blocks 
and determining the difference of luminance for each block instead of treating the 
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entire frame image at a same time. Alternatively, they can be reduced by 
determining the luminance histogram of each frame and subsequently the 
difference in the luminance histogram between two frames instead of determining 
the difference of luminance for each pixel. Still alternatively, they can be reduced 
by determining the average luminance for each frame image and subsequently the 
difference in the average luminance between two frames. To the contrary, if the 
storage capacity and the amount of arithmetic operations do not exceed respective 
permissible levels, the accuracy of detection can be improved by determining the 
difference of luminance and that of color histogram for individual component 
colors such as R (red), G (green) and B (blue). 
[160] 

Meanwhile, the audio signal 2b is digitized by A/D converter 13 and stored 
in audio signal buffer 14. The audio signal buffer 14 has a storage capacity for 
storing at least 2-channel stereophonic audio signals for left (L) and right (R) for a 
predetermined time period Ti (e.g., 30ms, which is to be referred to as a frame 
length hereinafter). The audio signals read out from the audio signal buffer 14 are 
then sent to amplitude detector 15, correlation detector 16 and spectrum detector 17 
[161] 

The amplitude detector 15 detects the mean square amplitude of a 
predetermined short period of time T 2 (e.g., 15ms, which is to be referred to as a 
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frame period hereinafter) from the audio signals stored in the audio signal buffer 14. 
More specifically, if the audio signal buffer 14 stores 2-channel stereophonic audio 
signals for left and right, the amplitude detector 1 5 detects the short period mean 
square amplitude of each predetermined time period T 2 (15ms, 1 frame period) 
from the 2-channel stereophonic audio signals S L [m] and S R [m]. Note that m (m 
= 0, M-l) represents the sample number in the buffer, showing a discretized 
time period, and the largest number M corresponds to a frame length TV 
[162] 

More specifically, the amplitude detector 15 calculates the mean square 
amplitude A[n] for the 2-channel audio signals for left and right in the n-th frame 
by using formula (15) below. 
[163] 

[Equation 11] 




(15) 



[164] 

The mean square amplitude A[n] obtained by, or the detection output of, 
the amplitude detector 15 is then sent to characteristic value buffer 18 as a 
characteristic value of the audio signal. 
[165] 
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The correlation detector 16 detects the correlation coefficient for the audio 
signal of each frame prior to the normalization, using the audio signals stored in the 
audio signal buffer 14 and also the short period energy to be used in the operation 
of normalization to be carried out downstream. If the audio signal buffer 14 stores 
2-channel stereophonic signals for left and right, the correlation detector 16 detects 
the correlation coefficient for the 2-channel audio signals for left and right for a 
frame prior to the normalization out of the 2-channel stereophonic signal for left 
and right read out from the audio signal buffer 14 and also the short period energy 
to be used in the operation of normalization to be carried out downstream. 
[166] 

More specifically, the correlation detector 16 calculates the correlation 
efficient A LR [n] of the 2-channel audio signals for left and right for the n-th frame 
by using formula (16) below. It also calculates the left channel audio signal 
energy A LL [n] by using formula (17) and the right channel audio signal energy 
ArrDi] by using formula (18) as shown below. 
[167] 

[Equation 12] 

u-\ 

A LR in]= HS L [m + nT 2 ]S R [m + nT 2 ] 

(16) 
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M-1 

A LL M=Y, s l[ m + nT 2] 

m =o (17) 

M-1 

A RrM = lL S l[ m+nT 2] 

m =o (18) 

[168] 

The detection output of the correlation detector 16 including the correlation 
coefficient A LR [n], the left channel audio signal energy Aix[n] and the right 
channel audio signal energy A^fn] are sent to the characteristic value buffer 1 8 as 
so many characteristic values. 
[169] 

The spectrum detector 17 calculates the short period spectrum by using the 
audio signals stored in the audio signal buffer 14. If the audio signal buffer 14 
stores 1 -channel stereophonic audio signals for left and right, the spectrum detector 
1 7 calculates the short period spectrum out of the 2-channel stereophonic audio 
signals S L [m] and S R [m] for left and right read out from the audio signal buffer 14. 
[170] 

More specifically, the spectrum detector 17 determines the discrete 
spectrum F[k;n] of the 2-channel audio signals for left and right of the n-th frame. 
If k = 0, K-l represents the discretized frequency, the discrete spectrum F[k;n] is 
expressed by formula (19) below. 
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[171] 

[Equation 13] 

F[k;n] = 

[172] 



(19) 



The arithmetic operation of formula (19) can be realized typically by means 
of an appropriate technique such as Fast Fourier Transform (FFT) or linear 
prediction analysis (LPC). 
[173] 

The computational output, or the short period discrete spectrum F[k:n] of 
the spectrum detector 17 is sent to the characteristic value buffer 18 as a 
characteristic value for the audio signal. 
[174] 

The broadcasting mode signal 2c is made to show a discretized value that 
conforms to the above described audio signal processing frame. 
[175] 

More specifically, the broadcasting mode signal 2c of the n-th frame may 
be made to show a numerical value B[n] as expressed by formula (20). 
[176] 

[Equation 14] 
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10 (monauralmode) 
1 (stereophonic mode) 
2 (sound - multiplex mode) 

[177] 

The numerical value B[n] obtained by discretizing the broadcasting mode 
signal 2c is then sent to the characteristic value buffer 1 8 as a characteristic value 
of the TV broadcast signal. 
[178] 

Similarly, temporal signal 3a is also made to show a discretized numerical 
value T[n] that conforms to the audio signal processing frame and sent to the 
characteristic value buffer 1 8 as a characteristic value. 
[179] 

Then, the characteristic value buffer 1 8 stores the characteristic value G[n] 
shown in formula (21) below that includes the detection output C[n] from the cut 
change detector 12, the means square amplitude A[n] from the amplitude detector 
15, the correlation coefficient A L R[n] and the audio signal energy values A LL [n] and 
ARR[n] from the correlation detector 16, the short period discrete spectrum F[k;n] 
from the spectrum detector 17, the discretized numerical value B[n] of the 
broadcasting mode signal 2c and the discretized numerical value T[n] of the 
temporal signal 3 a for a predetermined period of time T 3 . Note that the time T 3 is 
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so determined that the buffer can store at least a CM and may typically be 80 

seconds. 

[180] 

G[n] = {C[n], A[n], A LR [n], A LL [n] 5 A^n], F[k;n], B[n], T[n]} (21) 

The front end section of the CM detecting section 202 shown in FIG. 1 5 
comprises the above described components from the A/D converter 10 to the 
characteristic value buffer 18. Now, the operation of the front end section will be 
described by referring to the flow charts of FIGS. 16 and 17. Note that Steps S30 
through S32 in FIG. 16 are for the video signal 2a, whereas Steps S33 through S40 
in FIG. 17 are for the audio signal 2b, the broadcasting mode signal 2c and the 
temporal signal 3 a. 
[181] 

Firstly, the operation of processing the video signal 2a will be described by 
referring to FIG. 16. In Step S30, the front end section stores the video signal 2a 
for at least a frame that is digitized by the A/D converter 10 in the frame memory 
1 1 . The frame memory 11 is so designed as to handle a video signal 2a for a 
frame as a sample. Therefore, as the video signal 2a for a frame is input, the 
video signals 2a already stored in the frame memory 1 1 are shifted by a frame and 
the video signal 2a that was stored first in the frame memory 1 1 is pushed out as 
output. 
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[182] 

Next, the front end section reads out the video signal 2a from the frame 
memory 11 and transmits it to the cut change detector 12 in Step S31, which cut 
change detector 12 by turn produces the detection output C[n] in a manner as 
described above. 
[183] 

Then, the front end section stores the detection output C[n] in the 
characteristic value buffer 18 in Step S32. 
[184] 

Now, referring to FIG. 17 showing the operation of processing the audio 
signal 2b, the front end section inputs the audio signal 2b that is digitized by the 
A/D converter 13 into the audio signal buffer 14 and stores the audio signal 2b for 
at least a frame period T 2 in the audio signal buffer 14 in Steps S33 and S34. The 
audio signal buffer 14 is so designed as to handle an audio signal 2b for a frame 
period T 2 as a sample. Therefore, as the audio signal 2b for a frame period T 2 is 
input, the audio signals 2b already stored in the audio signal buffer 14 are shifted 
by a frame period T 2 and the audio signal 2b for a frame period T 2 that was stored 
first in the audio signal buffer 14 is pushed out as output. 
[185] 

As the audio signal buffer 14 stores at least an audio signal 2b for a frame 
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period T 2 , the front end section reads out the audio signal 2b stored in the audio 
signal buffer 14 and transmits it to the amplitude detector 15 in Step S3 5, which 
amplitude detector 15 by turn produces the means square amplitude A[n] in a 
manner as described above. 
[186] 

At the same time, the front end section transmits the audio signal 2b stored 
in the audio signal buffer 14 to the correlation detector 16 in Step S36, which 
correlation detector 16 by turn produces the correlation coefficient A LR [n] and the 
audio signal energy values A LL [n], A^n] in a manner as described above. 
[187] 

Concurrently, the front end section transmits the audio signal 2b stored in 
the audio signal buffer 14 to the spectrum detector 17 in Step S37, which spectrum 
detector 17 by turn produces the short period discrete spectrum F[k;n] in a manner 
as described above. 
[188] 

Furthermore, the front end section determines the discretized numerical 
value B[n] from the broadcasting mode signal 2c and the discretized numerical 
value T[n] from the temporal signal 3a in Step S38. 
[189] 

Then, the front end section stores the characteristic value G[n] including the 
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detection output C[n] from the cut change detector 12, the root-mean-square 
amplitude A[n] from the amplitude detector 15, the correlation efficient A L R[n] and 
the audio signal energy values A LL [n], ARR[n] from the correlation detector 16, the 
short period discrete spectrum F[k;n] from the spectrum detector 17, the discretized 
numerical value B[n] of the broadcasting mode signal 2c and the discretized 
numerical value T[n] of the temporal signal 3a in the characteristic value buffer 18. 
[190] 

Returning to FIG. 15, now the operation of the band end section will be 
described below. In the following description, n denotes the frame number of the 
characteristic value of each frame stored in the characteristic value buffer 18. The 
characteristic value of the most recent frame is expressed by G[0] and the value of 
n increases for an older frame so that, when the most recent frame is updated, the 
numbers of all the other frames and hence those of their characteristic values are 
shifted by 1). 
[191] 

Referring to FIG. 15, the characteristic values stored in the characteristic 
value buffer 18 are sent to CM candidate detector 19 on a frame by frame basis. 
[192] 

The CM candidate detector 19 computationally detects the CM section, if 
any, of each frame on the basis of the above described "indispensable 
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requirements" that are satisfied by almost all CMs. The indispensable 
requirements include that the audio signal of a CM shows a "low sound level" and 
hence the sound level of the audio signal is below a predetermined threshold level 
(to be referred to as sound level requirement hereinafter), that there is a "switched 
image" and hence the video signal shows an abrupt change or a uniform luminance 
in a frame (to be referred to as video requirement hereinafter) and that there is a 
"specified time length (few variations in time length)" and hence the temporal 
interval of two frames that satisfy both the sound level requirement and the video 
requirement agrees with the time length of the CM (to be referred to as time 
requirement hereinafter). More specifically, these requirements can be defined by 
formulas (22), (23) and (24), using the above described characteristic values; 
[193] 

A[0]<A thsd (22), 

C[0] = 1 (23) and 

A[nj] < A^, C[n,] = 1 or A[n 2 ] < A^, C[n 2 ] = 1 or 
A[n 3 ]<A thsd ,C[n 3 ] = l (24) 
where Athsd is the predetermined threshold value for the square-root amplitude and 
ni, n 2 and n 3 are respective numerical values obtained by reducing the time lengths 
specified as CM lengths (e.g., 15 seconds, 30 seconds and 60 seconds as described 
above for the purpose of this embodiment) in terms of the unit of frame period. 
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For the purpose of improving the practical applicability, the numerical values n h n 2 

and n 3 are allowed to be slightly variable. 

[194] 

Now, the operation of the CM candidate detector 19 will be described by 
referring to FIG. 18. 
[195] 

Referring to FIG. 18, like the frame memory and the audio signal buffer 
described above respectively by referring to Step S32 in FIG. 16 and Step S40 in 
FIG. 17, the characteristic value buffer 18 is so designed as to operates for 
inputting, shifting and outputting signals on the basis of a unit of a frame. In other 
words, the characteristic value buffer 18 is adapted to handle the characteristic 
value of a frame as a sample. Therefore, when the characteristic value of a frame 
is input, all the characteristic values already stored in the characteristic value buffer 
18 are shifted by 1 frame and the characteristic value of a frame that was stored 
first in the characteristic value buffer 1 8 is pushed out as output. 
[196] 

As a result of the processing operations of Steps S50 and S51 of inputting 
the characteristic value of a frame (a sample) from the characteristic value buffer 
18, the CM candidate detector 19 checks if the characteristic value of a frame (a 
sample) meets the indispensable requirements including the sound level 
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requirement, the video requirement and the time requirement in Steps S52 and S53. 
More specifically, the CM candidate detector 19 determines if the frame meets the 
indispensable requirements including the sound level requirement, the video 
requirement and the time requirement or not by comparing the means square 
amplitude A[0] of the first frame and the predetermined threshold value A thsd for 
square-root amplitude in Step S52 and checking if the detection output C[0] is 
equal to 1 or not in Step S53. If it is determined by the CM candidate detector 19 
that the means square amplitude A[0] does not exceed the predetermined threshold 
value A thsd for square-root amplitude and the indispensable requirements are met as 
a result of the processing operations of Steps S52 and S53, the frame is selected as 
CM candidate and the operation proceeds to Step S57 and on (Step S54 through 
S56 will be described hereinafter). If, on the other hand, it is determined by the 
CM candidate detector 19 that the means square amplitude A[0] exceeds the 
predetermined threshold value A thsd for square-root amplitude or the indispensable 
requirements are not met as a result of the processing operations of Steps S52 and 
S53, the frame is not selected as CM candidate and the operation returns to Step 
S50. 
[197] 

If it is determined that the means square amplitude A[0] does not exceed 
the predetermined threshold value Ath sd for square-root amplitude and the 
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indispensable requirements are met as a result of the processing operations of Steps 
S52 and S53, the CM candidate detector 19 retrieves the CM starting frame n s in 
Step S57 and also the CM ending frame n e in Step S58 and then computationally 
determines the CM starting time T s in Step S59 and also the CM length W in Step 
S60. 
[198] 

After the retrieving and computing operations of Steps S57 through S60, 
the CM candidate detector 19 refers to the CM candidate table, which will be 
described hereinafter, in Step S61 and, if it finds a CM candidate whose CM 
starting time T s and CM length T w respectively agree with those of the detected CM 
candidate, it returns to Steps S54 through 56. If, on the other hand, it does not 
find any such a CM candidate, it adds the CM candidate as new one and returns to 
Steps S54 through S56. 
[199] 

When the processing operations of Steps S54 through S56 are conducted 
for all the time lengths, the CM candidate detector 19 returns to Step S50 and 
repeats the above steps for the next input. 
[200] 

The CM starting frame n s refers to the frame number of the first frame 
whose means square amplitude A[n] exceeds the threshold value Ath sd for 
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square-root amplitude as viewed from the frame that meets the time requirements 
out of the frames expressed by rii, n 2 and n 3 toward the most recent frame. 
Similarly, the CM ending frame n e refers to the frame number of the last frame 
whose means square amplitude A[n] does not exceed the threshold value A ths d for 
square-root amplitude as viewed from the 0-th frame toward the older frames. 
Furthermore, the CM starting time T s can be obtained by using the formula of T s = 
T[n s ] and the number of the CM starting frame n s . Similarly, the CM length T w 
can be obtained by using the formula of T w = T[n e ] - T[n s ]. 
[201] 

FIG. 19 shows an example that can be used for defining the indispensable 
requirements. In the row of A[n] in FIG. 19, [o] denotes a frame whose means 
square amplitude does not exceed the threshold value for square-root amplitude 
Athsd and [x] denotes a frame whose root-mean-square amplitude exceeds the 
threshold value for square-root amplitude A thsd . In the illustrated example, A[0], 
C[0] and A[ni], C[ni] meet the requirements and the frame that is found left 
relative to ni and satisfies the requirement of A[n] = x is n s , whereas the last one of 
the successive frames found right relative to 0 and satisfy the requirement of A[n] = 
o is n e . 
[202] 

As a result of the above processing operation, the CM candidate detector 19 
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detects a CM candidate each time the characteristic value of a frame (sample) is 
input and, if a CM candidate is detected, it enters the candidate into the CM 
candidate table. 
[203] 

FIG. 20 shows an example of CM candidate table. Referring to FIG. 20, 
the CM candidate table contains items including the starting time T s , the length T w , 
the characteristic values Qi through Q n as determined by the additional 
requirement computing unit 20, which will be described hereinafter, and the score 
R and the result of score judgment Z determined by the additional requirement 
determiner 21, which will also be described hereinafter. As shown in FIG. 20, 
only the starting time T s and the length T w are written at the stage of the CM 
candidate table 19a prepared by the CM candidate detector 19. Thus, the CM 
candidate table is used to describe and manage the starting time T s and the length 
T w obtained by the CM candidate detector 19, the characteristic values Ch through 
Qn computed by the additional requirement computing unit 20 and the score R and 
the result of score judgment Z determined by the additional requirement determiner 
21 . The CM candidate table is maintained until the entry is judged to be a CM or 
not a CM and is output as CM detection output 4a from the rule judging unit 22 if it 
is judged to be a CM but simply discarded if it is judged to be not a CM. 
[204] 
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The CM candidate table 19a carrying only the starting time T s and the 
length T w as described by the CM is then sent to the additional requirement 
computing unit 20. 
[205] 

The additional requirement computing unit 20 extracts the characteristic 
values Q] through Q n from the candidate sections entered to the CM candidate 
table 19a, referring to the characteristic value buffer 18, in a manner as described 
below and adds them to the CM candidate table 19a to produce a CM candidate 
table 20a, which is then output to the additional requirement judging unit 21 . 
[206] 

FIG. 21 schematically illustrates how the characteristic values Qi through 
Qn are computationally determined by the additional requirement computing unit 
20. 
[207] 

Referring to FIG. 21, the horizontal axes represent the frame number 
(corresponding to the discrete time). In FIG. 21, (a) shows the cut change 
detection output C[n] and (b) shows the discretized numerical value B[n] of the 
broadcast mode signal 2c, whereas (c) shows the short period discrete spectrum 
S[k,n] of the audio signal and (d)shows the root-mean-square amplitude A[n] of the 
audio signal. The part indicated by ni (the part between the dotted lines) in FIG. 
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21 shows a CM candidate. In (a) of FIG. 21, the position indicated by CT shows 
the position where the cut change detection output C[n] is found to be equal to 1 
(and hence where a cut change is detected). In (b) of FIG. 2 1 , the part M indicates 
that it is in a broadcasting mode of some sort or another. In (c) of FIG. 21, SI, S2, 
S3 and S4 indicate that a spectrum component of some sort or another is there. In 
(d) of FIG. 21, AM indicates a change in the square-root amplitude and Ch through 
Qn indicates the respective positions where the characteristic values Qi through 
Qn are computationally determined by the additional requirement computing unit 
20. 
[208] 

Now, each of the characteristic values Ch through Qn that are 
computationally determined by the additional requirement computing unit 20 will 
be described below. 
[209] 

The characteristic value Qi is the front break length, or the length of the 
low sound level section (to be referred to as "front break section") located 
immediately before a CM candidate section, where A[n] is successively less than a 
predetermined threshold value A thsd . In FIG. 21, the section between the 
corresponding dotted broken lines represents the front break length Qi. 
[210] 
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The characteristic value Q 2 is the rear break length, or the length of the low 
sound level section (to be referred to as "rear break section") located immediately 
after a CM candidate section, where A[n] is successively less than a predetermined 
threshold value A thsd . In FIG. 21, the section between the corresponding dotted 
broken lines represents the rear break length Q 2 . 
[211] 

The characteristic value Q 3 is the front break minimal amplitude. The 
front break minimal amplitude Q 3 is the smallest value of A[n] in the front break 
section. 
[212] 

The characteristic values Q 4 is the rear break minimal amplitude. The rear 
break minimal amplitude Q 4 is the smallest value of A[n] in the rear break section. 
[213] 

The characteristic value Q 5 is the left/right correlation value. The left/right 
correlation value Q 5 is that of the 2-channel audio signals S L [m] and S R [m] for left 
and right. They can be computationally determined by using formula (25) below 
and A LR [n], A LL [n] and A^n] obtained by formulas (16) through (18). 
[214] 

[Equation 15] 
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(25) 

[215] 

In the computation of formula (25), the original waveform is partly added 
for a plurality of times as frames are made to overlap but this problem does not 
significantly affect the system. If the memory capacity and the processing speed 
are large enough for maintaining the original waveform, the operation of formula 

(25) can be replaced by the mutual correlation of the waveform. 
[216] 

The characteristic value Q 6 is the average amplitude. The average 
amplitude Q 6 is the RMS of the amplitudes of the audio signals (root-mean-square 
amplitude) of the CM candidate section. This can be obtained by using formula 

(26) below. 
[217] 

[Equation 16] 




[218] 

In the computation of formula (26), as in the case of the computation of the 
left/right correlation value, the original waveform is partly added for a plurality of 
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times as frames are made to overlap but this problem does not significantly affect 
the system. If the memory capacity and the processing speed are large enough for 
maintaining the original waveform, the operation of formula (25) can be replaced 
by that of RMS of the original waveform. 
[219] 

The characteristic value Q 7 is the number of cuts. The number of cuts Q 7 
is obtained by counting the number of cut changes (the number of CTs) in the CM 
candidate section. In other words, this is obtained by an operation of counting the 
number of times when C[n] = 1 occurs in section [n s , n e ). 
[220] 

The characteristic value Q 8 is the broadcasting mode. The broadcasting 
mode as used herein refers to the prevailing broadcasting mode in the CM 
candidate section. This is obtained by an operation of selecting the broadcasting 
mode Qg that appears most frequently in B[n] in [n s , n e ). 
[221] 

The characteristic value Q 9 is the number of adjacent candidates. The 
number of adjacent candidates Q 9 indicates that if the sounded sections found 
before and after the CM candidate are also CM candidates or not. The 
characteristic value Q 9 is equal to [2] when a CM candidate is found before and 
after the CM candidate and [1] when a CM candidate is found only before or after 
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the CM candidate. It is equal to [0] when no other CM candidate is found before 
and after the CM candidate. The operation of determining the number of adjacent 
candidates is realized by checking the CM candidate table. If there is a CM 
candidate that is immediately succeeding the CM candidate in question or not is 
determined by seeing if the sum of the starting time T s , the length T w and the rear 
break length Q 2 or (T s + T w + Q 2 ) agrees with the starting time (T' s ) of some other 
CM candidate or not. Similarly, if there is a CM candidate that is immediately 
preceding the CM candidate in question or not is determined by seeing if the 
difference of the starting time T s , and the front break length Ch or (T s - Qj) agrees 
with the sum of the starting time T' s and the length T' w or (T s + T' w ) of some other 
CM candidate or not. 
[222] 

The characteristic values Q 10 and Q n are the front spectrum difference 
energy and the rear spectrum difference energy. The front spectrum difference 
energy and the rear spectrum difference energy Q 10 and Qn are used to quantize the 
change in the sound quality at the boundary of a CM and the program containing 
the CM or at the boundary of a CM and another CM. These are defined as the 
square sum of the difference between the average spectrum values of the opposite 
sides of the boundary and obtained by using formulas (27) through (32) below; 
[223] 
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[Equation 17] 

SM = ^F[k;n e -n] 

N »-o (27), 

S 2[k] = ^rY,F[k;n s +n] 

N »-o (28), 

s 3 [*] = ^5M*;".-«] (29), 

N »-o (30), 

Q 10 =7i L L(5 2 U]-5 1 [^) 2 

no™ * (31) and 

^ * (32) 

[224] 

where N is the number of frames used for determining the average of spectrum 
values, n' e is the number of the ending frame of the sounded section found 
immediately before the CM candidate section (see FIG. 21), n' s is the number of 
the starting frame of the sounded section found immediately after the CM candidate 
section, Sj[k] is the average spectrum immediately before the end of the sounded 
section that is found immediately before the CM candidate section, S 2 [k] is the 
average spectrum immediately after the start of the CM candidate section, S 3 [k] is 
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the average spectrum immediately before the end of the CM candidate section, 
S 4 [k] is the average spectrum immediately after the start of the sounded section that 
is found immediately after the CM candidate section and S norm is an appropriate 
normalization constant. 
[225] 

The above described additional requirement computing unit 20 additionally 
describes the characteristic values Ch through Qn that are determined in a manner 
as described above on the CM table 19a to produce the CM candidate table 20a as 
output. The CM candidate table 20a is sent output the additional requirement 
judging unit 21. 
[226] 

The additional requirement judging unit 21 receives the CM candidate table 
20a as input and calculates the score R for the CM candidate by performing a 
parameter transform on the characteristic values of the CM candidate typically, 
using a threshold function, and subsequently weighting the obtained values so that 
the CM candidate is judged to be a strong candidate when R exceeds a 
predetermined threshold value. The additional requirement judging unit 21 
additionally describes the score R and the result of score judgment Z to the CM 
candidate table 20a to produce the CM candidate table 21a as output. 
[227] 
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FIG. 22 schematically illustrates the configuration of the additional 
requirement judging unit 21 . 
[228] 

Referring to FIG. 22, the characteristic values Qi through Q L of the CM 
candidate table 21a are sent to respective function computing units 50] through 50 L , 
which perform transform operations by using respective parameter transform 
functions H^) through H L () and then obtain the products of the outcomes of the 
respective computations and respective weights W] through W L by means of 
respective weighting units 51 1 through 51 L . The characteristic values obtained as 
a result of the weighting operations, using the respective weighting units 51] 
through 51 L , are added by summation adder 52 to obtain the score R. The score R 
output from the summation adder 52 is compared with a predetermined threshold 
value by score judging unit 53 and, if the score R exceeds a predetermined 
threshold value, an output telling that the CM candidate is judged to be a strong 
CM candidate is produced. The CM candidate that is determined by the score 
judging unit 53 so as to be not exceeding the threshold value is erased from the 
table. 
[229] 

More specifically, the additional requirement judging unit 21 computes the 
score, using formula (33) below; 



86 



[230] 

[Equation 1 8] 

K = i>,ff/(e,) 

(33) 

[231] 

where Hi() is the parameter transform function predetermined for each of the 
characteristic values, Wi is the predetermined weight and L is the number of 
characteristic values (=11). Note that 1 represents a number between 1 and 1 1 . 
[232] 

The parameter transform function Hi() that is used by the function 
computing units 50i through 50 L may be a rectangular function, which is the most 
simple function. For example, rectangular function Rect(x; t], t 2 ) as shown in (b) 
of FIG. 23 may be used and t n and t 2 i may be selected for the lower and upper 
limits of the reference value that is predetermined for each of the characteristic 
values. Then, if formula (34) is used, 1 is given when Ch is found within the 
scope of the reference value, whereas 0 is given when Ch if found out of the scope 
of the reference value. 
[233] 

H,(Q 1 ) = Rect(Q 1 ;t 1 ,t 2 ) (34) 

If a smooth transition from 0 to 1 or vice versa is to be realized near each of 
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the boundaries, a sigmoid function Sigm(x; t l5 t 2 ) as expressed by formula (35) 

below may be used. 

[234] 

[Equation 19] 

1 1 



1 + exp(-(x - t u )/ a u ) ' 1 + exp((* - t 2l )/ a 2l ) 
(35) 



[235] 

In FIG. 23, (c) schematically shows such smooth transitions. In FIG. 23, 
an and c 2 i are constants defining the profile of transition that are to be selected in 
advance typically by referring to the distribution of characteristic value. 
[236] 

While the weight W] applied by the weighting units 5 1 1 through 5 1 L may be 
artificially selected in advance on the basis of the statistic properties of the 
characteristic values, it is also possible to select the weight by making the device 
learn known learning samples typically by means of a neural network (see, inter 
alia, Nakagawa, "Pattern Information Processing", Maruzen (1999). In formula 
(35), 1 represents any number between 1 and 11. 
[237] 
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The operation of score judgment of said judging unit 53 is performed by 
processing the threshold value for score R, using formula (36) below; 
[238] 

Z = Unit ® - t r ) (36) 
where Unit (x) is a unit step function whose value is 1 when x > 0 and 0 when x < 0 
as shown in (a) of FIG. 23 and t is a judgment threshold value that is predetermined 
or automatically selected as a result of a learning process. 
[239] 

Then, the rule judging unit 22 receives the CM candidate table 21a obtained 
as a result of the score judgment of the additional requirement judging unit 21 as 
input and produces the CM starting time and the CM length as final CM detection 
output 4a that is obtained as a result of a predetermined rule judging operation as 
will be described hereinafter. That is, the rule judging unit 22 effects rule 
processing, thus selecting the most prominent CM candidate if there are two or 
more CM candidates at the same time. (The event that two or more CM 
candidates coexist shall be referred to as "competing relationship".) 
[240] 

Now, the operation of the rule judging unit 22 will be described by 
referring to the flow chart of FIG. 24. 
[241] 
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Firstly, in Step S70, the rule judging unit 22 selects a CM candidate for 
judgment from the CM candidate table. The selected CM candidate is the oldest 
candidate in the CM candidate table that has survived for a predetermined time 
period T 4 , which is long enough for containing several CMs and may typically be 
150 seconds. 
[242] 

Subsequently, in Step S71, the rule judging unit 22 checks the CM 
candidate table to see if any other CM candidate is found in the selected CM 
candidate section (which is the period from T s to T s + T w ). If it is determined in 
Step S71 that there is not any other CM candidate in the selected CM candidate 
section (No), the CM candidate is output as CM detection output and erased from 
the CM candidate table. 
[243] 

If, on the other hand, it is determined in Step S71 that there is another CM 
candidate in the selected CM candidate section (Yes), it is so judged that they are 
competing and the minimum length prioritizing rule is firstly applied to them in 
Step S72. With the minimum length prioritizing rule, the time section comprising 
a shorter CM candidate is prioritized among a number of time sections having a 
uniform length and comprising one or more than one CM candidates with different 
time lengths. More specifically, if there are a time section having a length of 30 
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seconds and comprising a single 30 seconds long CM candidate and a time section 
having a length of 30 seconds and comprising two 15 seconds long CM candidates, 
the 15 seconds long CM candidates are selected and the 30 seconds long CM 
candidate is discarded. 
[244] 

The minimum length prioritizing rule will be described further by referring 
to FIG. 25 that illustrates an example. 
[245] 

In FIG. 25, (a) shows a section where four CMs including CM1 through 
CM4 are successively broadcast, whereas (b) shows that eight CM candidates A 
through H are found in the CM candidate table. 
[246] 

Firstly, as the CM candidate of A is subjected to judgment as shown in (c) 
of FIG. 25, E and H compete with A. However, since section E can be described 
by A and B and section H can be described by A, B, C and D, they are discarded 
and hence A is selected. Subsequently, as the CM candidate of B is subjected to 
judgment as shown in (d) of FIG. 25, F appears as competing section (note that E 
and H have already been discarded because A is adopted). However, since section 
F can be described by B and C, it is discarded and hence B is adopted. Similarly, 
as the CM candidate of C is subjected to judgment as shown in (e) of FIG. 25, G 
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appears as competing section. However, since section G can be described by C 
and D, it is discarded and hence C is adopted. Finally, as the CM candidate of D 
is subjected to judgment as shown in (f) of FIG. 25, there is no longer any 
competitor and it is no longer necessary to apply the rule. Therefore, D is 
adopted. 
[247] 

As a result of the above operation, A, B, C and D are selected from the time 
section as CM candidates. If there arises a competing situation to which this rule 
cannot be applied, all the CM candidates are left in the table and the processing 
operation will simply be terminated. 
[248] 

Returning to FIG. 24, after the judgment in Step S72, the operation of the 
rule judging unit 22 proceeds to Step S73, where the rule judging unit 22 
determines if the CM candidate being subjected to judgment is discarded or not as a 
result of the application of the minimum length prioritizing rule. If it is 
determined in Step S73 that the CM candidate being subjected to judgment is 
discarded (Yes), the rule judging unit 22 erases the candidate from the CM 
candidate table and returns to Step S70. If, on the other hand, it is determined in 
Step S73 that the CM candidate being subjected to judgment is not discarded (No), 
the rule judging unit 22 checks the CM candidate table in Step S74 to see if there is 
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any other CM candidate found in the time section of the CM candidate that is being 

subjected to judgment. 

[249] 

If it is determined in Step S74 that there is not any other CM candidate (No), 
the rule judging unit 22 outputs the CM candidate being subjected to judgment as 
detection output and erase it from the CM candidate table in Step S80. If, on the 
other hand, it is determined in Step S74 that there is at least another CM candidate 
(Yes), the rule judging unit 22 proceeds to Step S75. 
[250] 

In Step S75, the rule judging unit 22 applies the adjacency prioritizing rule 
to the CM candidate. With the adjacency prioritizing rule, if there are a plurality 
of CM candidates that are competing with each other, priority is given to the CM 
candidate having adjacent CM candidates that immediately precedes and succeeds 
it. 

[251] 

The adjacency prioritizing rule will be described further by referring to FIG. 
26, illustrating an example. 
[252] 

In FIG. 26, (a) shows that four CMs, or CM 11 through CM 14, are 
broadcast successively in a time section and (b) shows that there exist a total of six 
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candidates, or candidates I through N, for the time section. The candidates M and 
N are selected as candidates simply because they incidently contains cut changes 
and low sound levels. It will be appreciated that, while such candidates are falsely 
selected for the time section, they survived the score judgment process for the 
additional requirements that is conducted to judge if the candidates are probably 
CMs or not because they contain CMs. 
[253] 

In the above example, the oldest CM candidate I is firstly subjected to 
judgment as shown in (c) of FIG. 26. While M competes with I, I has an adjacent 
candidate J but M does not have any adjacent candidate so that I is adopted and M 
is discarded. Then, the CM candidate J is subjected to judgment as shown in (d) 
of FIG. 26. While N completes with J, J has an adjacent candidate I and K but N 
does not have any adjacent candidate so that J is adopted and N is discarded. 
Subsequently, since the remaining CM candidates K, L do not have any competitor 
as shown in (e) and (f) of FIG. 26, both K and L are adopted without problem. 
[254] 

As a result of the above processing operation, I, J, K and L are selected as 
CM candidates from the time section of FIG. 26. 
[255] 

When none of the competing candidates have any adjacent candidate and 
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when a plurality of candidates have respective adjacent candidates, none of them 

are discarded and left in the CM candidate table. 

[256] 

Returning to FIG. 24, after the operation of Step S75, the rule judging unit 
22 proceeds to Step S76, where the rule judging unit 22 determines if the CM 
candidate being subjected to judgement is discarded or not as a result of the 
application of the adjacency prioritizing rule. If it is determined in Step S76 that 
the CM candidate being subjected to judgment is discarded (Yes), the rule judging 
unit 22 erases the candidate from the CM candidate table and returns to Step S70. 
If, on the other hand, it is determined in Step S76 that the CM candidate being 
subjected to judgment is not discarded (No), the rule judging unit 22 checks the 
CM candidate table in the next Step S77 to see if there is any other CM candidate 
found in the time section of the CM candidate that is being subjected to judgment. 
[257] 

If it is determined in Step S77 that there is not any other CM candidate (No), 
the rule judging unit 22 outputs the CM candidate being subjected to judgment as 
detection output and erase it from the CM candidate table in Step S80. If, on the 
other hand, it is determined in Step S77 that there is at least another CM candidate 
(Yes), the rule judging unit 22 proceeds to Step S78. Where the rule judging unit 22 
applies the score prioritizing rule to the CM candidate. With the score prioritizing 
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rule, if the competing situation is not dissolved as a result of the application of the 
above rules, the candidate having a higher score R given as a result of the score 
judgment of the additional requirement judging unit 21 has priority. This score 
prioritizing rule is repeatedly applied until the competing situation is dissolved. 
[258] 

The score prioritizing rule will be described by referring to FIG. 27, 
illustrating an example. 
[259] 

In FIG. 27, (a) shows that four CMs, or CM 21 through CM 24, are 
broadcast successively in a time section and (b) shows that there exist a total of 
seven candidates, or candidates P through W, for the time section. 
[260] 

In the above example, the oldest CM candidate P is firstly subjected to 
judgment as shown in (c) of FIG. 27 and competing with U. The competing 
situation is not dissolved by applying the minimum length prioritizing rule and the 
adjacency prioritizing rule. 
[261] 

Therefore, all the competing situations related to the competing candidates 
are retrieved from the CM candidate table. Hence there are six competing 
situations (P-U), (U-Q), (Q-V), (V-R), (R-W) and (W-S), involving seven 
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candidates, and the candidate having the highest score is selected by applying the 
score prioritizing rule. In the above example, R has the highest judgment score of 
(2.0) and therefore R having this score is adopted and the candidates V, W that are 
competing with R are discarded as shown in (d) of FIG. 27. 
[262] 

However, as shown in (e) of FIG. 27, the completing relationship of (P-U) 
is not dissolved by the above described adoption of R. Therefore, again, all the 
competing situations related to the competing candidates are retrieved from the CM 
candidate table. Since V is discarded, there are only two competing situations 
(P-U) and (U-Q), involving three candidates. 
[263] 

Of the above candidates, the candidate Q having the highest judgment score 
of (1.9) is adopted and the candidate U that is competing with Q is discarded as 
shown in (f) of FIG. 27. 
[264] 

With the above processing operation, there does not remain any competing 
situation, involving P, so that P is adopted. Additionally, U, V and W are 
discarded, while Q, R and S are adopted. 
[265] 

It will be appreciated that, if all the related competing situations are not 
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retrieved and the score prioritizing rule is applied only to the competing situations 
of the candidates being judged, firstly U is adopted and P is discarded. 
Subsequently, because of the competing situation of U and Q, U that is once 
adopted is also discarded. Therefore, the rule judging unit 22 retrieves all the 
related competing situations in order to eliminate the risk of erroneously and 
incidentally discarding proper candidates because of the processing sequence that is 
determined only by chance. 
[266] 

Thus, all the competing situations of the selected candidates can be 
eliminated by the application of the score prioritizing rule. 
[267] 

Returning to FIG. 24, after the processing operation of Step S78, the rule 
judging unit 22 proceeds to Step S79, where the rule judging unit 22 determines if 
the candidates being subjected to judgement is discarded or not as a result of the 
application of the score prioritizing rule. If it is determined in Step S79 that the 
CM candidate being subjected to judgment is discarded (Yes), the rule judging unit 
22 erases the candidate from the CM candidate table and returns to Step S70. If, 
on the other hand, it is determined in Step S79 that the CM candidate being 
subjected to judgment is not discarded (No), the rule judging unit 22 outputs the 
starting time and the length of the CM as CM detection output in Step S80 and 
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erases the CM candidate from the CM candidate table before it returns to Step S70. 
[268] 

As described above, the above described first example of CM detecting 
section 202 of this embodiment firstly deterministically extracts CM candidates 
from the program on the basis of the indispensable requirements that almost all 
CMs satisfy, selects CM candidates by statistically evaluating the characteristic 
values of each CM candidate on the basis of the additional requirements that are 
used to indicate probable CMs and then dissolves any overlapping relationship of 
candidates on the basis of the logical requirement in order to accurately detect CMs. 
While this embodiment is adapted to a video/audio recording apparatus adapted to 
currently prevalent analog TV broadcasting, it will be appreciated that the CM 
detecting section 202 is applicable to digital TV broadcasting. It will be 
appreciated that, when the embodiment is applied to radio broadcasting, the part of 
the CM detecting section 202 for processing video signals can be omitted. 
[269] 

Now, a second example of CM detecting section 202 of the embodiment of 
the present invention will be described below. 
[270] 

FIG. 28 illustrates in detail the configuration of the second example of CM 
detecting section 202 of the embodiment of the invention. 
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[271] 

This example of CM detecting section 202 is adapted to handle only the 
basic ones of the above described additional requirements. More specifically, of 
the additional requirements 1) through 14), 11) through 14) are not used for the CM 
detecting section 202 to simplify the configuration thereof, (preventing the 
intricacy) 
[272] 

Like the first example of CM detecting section 202 shown in FIG. 15, this 
second example of CM detecting section 202 comprises a front end section and a 
rear end section. In FIG. 28, the components that are same as their counterparts of 
FIG. 15 are denoted respectively by the same reference symbols and will not be 
described any further. 
[273] 

Referring to FIG. 28, therefore, only the additional components (101, 102, 
103) and the extended function of the additional requirement computing unit 20 
will be described below. 
[274] 

The sound source identifying unit 101 of the front end section receives a 
digitized and framed audio signal 2b as input and outputs the sound source name of 
the frame of the audio signal 2b. The sound source may be voice, music, voice 
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and music and others. Any of the technologies illustrated in Kohchi et al. 5 
"Automatic Classification of Broadcast Sounds on the Basis of VQ distortions", 
Journal of the Society of Telecommunication Technology, DSP97-95/SP97-50, 
43/48 (1998), Minami et al., "Video Indexing and Applications thereof Using 
Sound Information", Paper of the Society of Telecommunication Technology, Vol. 
J81-D-II, No. 3, 529/537 (1998) and the specification and the drawing of Abe's 
Japanese Patent Application No. 11-190693 may be utilized for the purpose of the 
identifying the sound source of the input audio signal. 
[275] 

A numerical value is assigned to each of the sound source names identified 
by the sound source identifying unit 101. The numerical values that can be used 
for the purpose of the invention may include voice = 1 and music =2. The 
numerical value of each sound source is then input to the characteristic value buffer 
18 as characteristic value U[n]. 
[276] 

The program category data or the program category identifying unit 102 
arranged at the front end section is used to output the category name of the program 
being processed. The program category names that can be used for the purpose of 
the invention include news, drama, baseball and soccer. Program category data 
may be obtained from a TV program guide or a TV program table. Nowadays, 
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program category data can be obtained automatically by way of Internet. 
Alternatively, it is also possible to use a device adapted to automatically identify 
the category of the program being broadcast on the basis of the audio and video 
signals of the program without resorting to external information. The technology 
illustrated in the specification and the drawing of Abe's Japanese Patent 
Application No. 11-190693 may be utilized for the purpose of the identifying the 
program category form the audio and video signals of the program being broadcast. 
[277] 

A numerical value is assigned to each of the program category names 
identified by the program category data or the program category identifying unit 
102 on a frame by frame basis. The numerical values that can be used for the 
purpose of the invention may include news=l and drama=2. The numerical value 
of each program category is then input to the characteristic value buffer 1 8 as 
characteristic value W[n]. 
[278] 

Otherwise, the front end section has a configuration same as that of FIG. 

15. 
[279] 

In the case of the above described second example, the front end section is 
made to comprise the sound source identifying unit 101 and the program category 
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data or the program category identifying unit 1 02 so that the characteristic values 
U[n] and W[n] obtained by them are stored in the characteristic value buffer 18. 
Thus, the characteristic value G[n] of formula (21) stored in the characteristic value 
buffer 18 is expanded to formula (37) below. 
[280] 

G[n] ={C[n], A[n], A LR [n], A LL [n], A^n], F[k;n], 

B[n], T[n], U[n], W[n]} (37) 

While the rear end section of the CM detector 19 of this example is exactly 
same as that of FIG. 15, the CM candidate tables 19a through 21a of this second 
example are extended in a manner as described below. In the case of this second 
example, the CM candidate tables 19a through 21a are extended in such a way that 
the characteristic values Q 12 through Q 15 to the above described characteristic 
values Qi through Q u as shown in FIG. 29. Note that, in FIG. 29, the 
characteristic values Qi through Qu are omitted and not shown. 
[281] 

The CM probability data base 103 of the rear end section stores in advance 
the probability of transmission of each CM obtained as a function of time slot and 
the probability of transmission of each CM obtained as a function of program 
category and the elapsed time of program. The probabilities are read out from the 
CM probability data base 103 by referring to the current time and input to the 
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additional requirement computing unit 20. The probability data stored in the CM 
probability data base 103 can be obtained by statistically processing the actually 
broadcast programs. 



Thus, the additional requirement computing unit 20 of the second example 
is extended to carry out the operations of computing the characteristic values Q ]2 
through Qi 5 to the characteristic values Qi through Qn. 



The characteristic value Q n is obtained by checking if the CM candidate 
being examined contains one or more than one voice sections or not. The 
characteristic value Q 12 that shows if the CM candidate contains one ore more than 
one voice sections or not is determined by using formula (38) below. 
[284] 

[Equation 20] 



Like the presence or absence of voice, the characteristic value Q 13 is 
obtained by checking if the CM candidate being examined contains one or more 
than one music sections or not. The characteristic value Q 13 that shows if the CM 



[282] 



[283] 




(38) 



[285] 
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candidate contains one ore more than one music sections or not is determined by 

using formula (39) below. 

[286] 

[Equation 21] 



The characteristic value Q 14 is the probability of occurrence of the CM in 
question as determined on the basis of the current time (time slot probability). 
The additional requirement computing unit 20 uses the probability of broadcasting 
the CM obtained from the CM probability data base 103 directly as the 
characteristic value Q 14 . 



The characteristic value Q 15 is the probability of occurrence of the CM in 
question as determined on the basis of program category of the current program and 
the elapsed time from the start of the program (program category probability). 
The additional requirement computing unit 20 uses the probability of broadcasting 
the CM obtained from the CM probability data base 103 directly as the 
characteristic value Q 15 . 




(39) 



[287] 



[288] 



[289] 
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The arrangement of the CM detecting section 202 downstream to the 
additional requirement judging unit 21 is same as that the CM detecting section 202 
of FIG. 15 except that the variables are extended to include the characteristic values 
Q12 through Qi 5 and therefore will not be described here any further. 
[290] 

As a result of the above described extension, the CM detecting section 202 
of this example can detect CMs on the basis of the sound sources of the broadcast 
signal, taking the current time and the program category into consideration. 
[291] 

FIG. 30 is a schematic block diagram of the hardware components of the 
CM detecting section 202 of the embodiment that can be used for the purpose of 
the invention and illustrated in FIGS. 15 and 28. 
[292] 

Referring to FIG. 30, the AID converter 40 has the function of the AID 
converters 10 and 13 of FIGS. 15 and 28 and the memory 41 has the function of the 
frame memory 1 1 and that of audio signal buffer 14. 
[293] 

The AN processor, or DSP (digital signal processor) 42 has the functions 
of the cut change detector 112, the amplitude detector 15, the correlation detector 
16, the spectrum detector 17, the sound source identifier 101 and so on and the 
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memory 43 has the function of the characteristic value buffer 18. 
[294] 

The processor 44 has functions of the CM candidate detector 19, the 
additional requirement computing unit 20, the additional requirement judging unit 
21, the rule judging unit 22, the CM probability data base 103 and so on. 
[295] 

Either the A/V processor, or the DSP (digital signal processor) 42 or the 
processor 44 may be made to have the function of the operation control section 23. 
[296] 

[Effect of the Invention] 

In short, according to the invention, since the first signal section is detected 
out of an input signal and then extracted and recorded according to the result of the 
detection of the first signal section, it is now possible to detect a CM as the first 
signal out of a broadcast signal, or the input signal, so that the CM can be stored, 
viewed/listened to and/or retrieved without problem. 
[Brief Description of the Drawings] 

FIG. 1 is a schematic block diagram of a preferred embodiment of 
broadcast signal processing device according to the invention; 

FIG. 2 is a schematic block diagram of a subset of the embodiment 
including the CM detecting section, the CM extracting section and the CM 



107 



recording section; 

FIG. 3 is a flow chart of the operation of the subset of FIG. 2; 

FIG. 4 is a schematic illustration of discretization of an input audio signal 
and an input video signal and processing frames to be used for the signals; 

FIG. 5 is a schematic block diagram of a subset of the embodiment 
including the CM detecting section, the CM extracting section, the CM 
characteristics extracting section and the CM recording section; 

FIG. 6 is a flow chart of the operation of the subset of FIG. 5; 

FIG. 7 is a schematic illustration of an image that can be displayed on the 
display screen of the display section of the embodiment; 

FIG. 8 is a schematic block diagram of a subset of the embodiment 
including the CM recording section, the CM index generating section, the CM 
viewing section and the CM selecting section; 

FIG. 9 is a flow chart of the operation of the subset of FIG. 8; 

FIG. 10 is a schematic block diagram of a subset of the embodiment 
including the CM recording section and the characteristics comparing section, 

FIG. 1 1 is a flow chart of the operation of the subset of FIG. 10; 

FIG. 12 is a schematic illustration of another image that can be displayed 
on the display screen of the display section of the embodiment (to show retrieval 
icons); 
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FIG. 13 is a schematic block diagram of a subset of the embodiment 
including the CM recording section, the characteristics comparing section, the CM 
index generating section, the CM viewing section and the CM selecting section; 

FIG. 14 is a flow chart of the operation of the subset of FIG. 13; 

FIG. 15 is a schematic block diagram of the CM detecting section, 
illustrating a possible first configuration thereof; 

FIG. 16 is a flow chart of the operation of the video signal processing 
operation of the front end part of the CM detecting section of the embodiment; 

FIG. 17 is a flow chart of the operation of the audio signal processing 
operation of the front end part of the CM detecting section of the embodiment; 

FIG. 18 is a flow chart of the operation of the CM candidate detector of the 
CM detecting section of the embodiment; 

FIG. 19 is a schematic illustration of the operation of computationally 
determining the requirements to be met for the embodiment; 

FIG. 20 is a schematic illustration of a first possible CM candidate table 
that can be used in the CM detecting section of the embodiment; 

FIG. 21 is a schematic illustration of the operation of computationally 
determining characteristic values of the additional requirement determining 
computer of the CM detecting section of the embodiment; 

FIG. 22 is a schematic circuit diagram of the additional requirement 
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determining computer of the embodiment; 

FIG. 23 is a schematic illustration of a unit step function, a rectangular 
function and a sigmoid type function that can be used for the operation of 
computationally determining scores in the embodiment; 

FIG. 24 is flow chart of the operation of the rule determiner of the 
embodiment; 

FIG. 25 is a schematic illustration of a minimum length prioritizing rule 
that can be used for the purpose of the invention; 

FIG. 26 is a schematic illustration of an adjacency prioritizing rule that can 
be used for the purpose of the invention; 

FIG. 27 is a schematic illustration of a score prioritizing rule that can be 
used for the purpose of the invention; 

FIG. 28 is a schematic block diagram of the CM detecting section, 
illustrating a possible second configuration thereof; 

FIG. 29 is a schematic illustration of a second possible CM candidate table 
(extended part only) that can be used in the CM detecting section of the 
embodiment; and 

FIG. 30 is a schematic block diagram of the hardware components of the 
CM detection of the embodiment that can be used for the purpose of the invention. 
[Explanation of Referenced Numerals] 
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201 CM extracting section; 202 CM detecting section; 203 CM characteristics 
extracting section; 204 characteristics comparing section; 205 CM recording 
section; 206 CM index generating section; 207 CM selecting section; 208 CM 
viewing section; 801 display processor; 802 display section; 810 CM index images, 
or the video images; 81 1 CM reproduction icon; 812 sound reproduction icon; 813 
preceding page icon; 814 succeeding page icon; 815 retrieval icon 
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[Name of Document] ABSTRACT 

[Summary] 

[Task] 

A signal processing device and a signal processing method can accurately 
detect CMs out of broadcast signals and also be used for storing, accessing, 
retrieving and viewing/listening to CMs. 
[Means for Solution] 

CM detecting section (202) detects CM sections out of a broadcast signal. 
CM extracting section (201) extracts the signal part for the CM section from the 
broadcast signal on the basis of the CM section detection signal (202a) of the CM 
detecting section (202) and CM characteristics extracting section (203) extracts the 
characteristic value of the CM. Then, CM recording section (205) records the 
signal of the CM section and the characteristic value. CM index generating section 
(206) generates CM index information by using the signal of the CM section and 
the characteristic value and characteristics comparing section(204) determines 
agreement/disagreement of CMs. CM viewing section (208) displays CM retrieval 
information and reproduce the CM signal from the CM recording scanning (205) in 
response to the instruction of the user (209). As a result, the user (209) can view, 



112 



listen to and retrieve the desired CM. 
[Selected Drawing] FIG. 1 
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